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ABSTRACT 


This  report  deals  with  the  stability,  stabilization  and  optimi- 
zation of  systems  which  can  be  meaningfully  viewed  as  an  interconnection 
of  smaller  subsystems.  We  consider  systems  whose  properties  can  be 
determined  by  examining  the  corresponding  properties  of  the  subsystems 
and  their  interconnections  rather  than  the  system  as  a whole.  The  rele- 
vance of  such  systems  to  any  theory  of  large  scale  systems  is  evident. 

Heuristically  speaking,  a weak  interconnection  of  dynamical 
systems  should  be  analyzable  in  such  a segmented  fashion.  We  present 
some  formulations  of  the  weak  interconnection  concept  in  Section  II  in 
the  context  of  stability  studies  of  economic  models.  It  is  shown  how  the 
modelling  of  an  economy  requires  the  consideration  of  systems  formed 
by  suitably  bounded  interconnections  of  stable  subsystems.  Lyapunov's 
second  method  is  employed  to  unify  numerous  results  as  well  as  to 
provide  interesting  extensions. 

In  contrast  to  Section  II  where  weakly  interconnected  stable 
systems  follow  from  modelling  requirements,  Section  IV  considers  the 
interconnection  of  stable  dynamic  systems  through  additive  terms  and 
investigates  under  what  conditions  cn  these  additive  terms  stability  will 
be  preserved.  The  methods  of  dissipative  system  theory  are  used  to 
derive  such  conditions,  the  attractive  feature  of  which  is  that  they  can 
be  used  to  design  decentralized  feedback  stabilizing  controllers. 

The  problem  of  decentralized  stabilization  is  reduced  to  an  infinite 
duration  linear  quadratic  game,  a complete  solution  of  which  is  given  in 
Section  V.  As  expected,  Riccati-type  equations  play  an  important  role  in 
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such  problems.  This  role  is  clarified  in  Section  V. 

Section  III  deals  with  the  comparison  of  the  solutions  of  optimal 
control  problems  for  interconnected  systems.  The  theory  of  dynamic 
programming  and  the  Hamilton  Jacobi  Bellman  equation  is  the  main  tool 
used.  In  addition,  the  same  theory  is  applied  to  an  extensive  study  of 
optimization  algorithms,  the  main  result  being  the  extension  of  the 
applicability  of  the  well-known  Kleinman  algorithm  for  the  Riccati 
equation  to  cover  the  ones  encountered  in  Sections  IV  and  V.  This 


result  provides  an  important  ingredient  for  any  practical  application  of 
the  decentralized  stabilization  procedures  proposed  in  these  Sections. 
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I. 


SCOPE  OF  THE  PAPER 


This  paper  examines  some  problems  arising  in  the  analysis  of 
dynamical  systems  whose  large  dimensionality  makes  the  application  of 
standard  analysis  techniques  infeasible  due  to  excessive  computational 
requirements.  Such  a system  will  be  referred  to  as  a large  scale 
system.  A theory  of  large  scale  systems  (1.  s.  s.  t)  tries  to  identify 
classes  of  large  systems  whose  properties  can  be  efficiently  determined. 
Its  scope  is  thus  twofold,  first  to  isolate  classes  of  large  systems  which 
are  useful  for  applications  and  second  to  propose  efficient  algorithms 
for  their  analysis. 

A large  system  structure  often  encountered  in  practice  is  that  of 
an  interconnected  system,  where  a number  of  dynamical  subsystems 
interact  to  form  a large  system.  An  electric  power  system  is  an  example 
where  the  subsystems  and  the  interconnections  can  be  physically 
identified.  It  would  be  desirable  to  develop  methods  for  determining  the 
properties  of  the  interconnected  system  by  somehow  aggregating  the 
corresponding  properties  of  the  subsystems.  Such  a task  perhaps  too 
ambitious  as  the  effect  of  the  interconnections  can  be  very  hard  to  assess. 
It  might  be  claimed,  from  a heuristic  point  of  view,  that  of  the  subsystems 
are  interconnected  weakly  it  would  be  possible  to  study  the  whole  system 
by  aggregating  the  properties  of  the  subsystems. 

It  is  not  obvious  how  one  should  define  the  term  "weak  inter- 
connection. " To  define  a system  as  weakly  interconnected  when  its 
analysis  can  be  done  by  aggregating  its  subsystem  properties  would  be 
circular.  In  Sections  II  and  IV  we  study  some  non-circular  formulations 
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of  this  concept  as  applied  to  the  stability  and  stabilization  of  inter- 
connected systems.  An  examination  of  the  meaning  of  economic  stability 
carried  out  in  Section  II  results  in  the  concept  of  Metzler  stability  or 
stability  independent  of  adjustment  rates:  Economic  processes  cannot  be 
adequately  modelled  by  a single  differential  equation  for  the  simple 
reason  that  the  rates  at  which  prices  adjust  to  a disequilibrium  are 
varying  in  an  unpredictable  way.  It  is  meaningful  to  refer  to  an  economic 
model  as  stable  if  this  property  holds  independent  of  adjustment  rates. 

The  implications  of  Metzler  stability  are  explored  in  Section  II  by 
Lyapunov  methods  which  provide  a unified  setting  for  known  results  on 
linear  systems  as  well  as  their  generalization  to  nonlinear  ones.  Most 
of  the  conditions  thus  derived  can  be  interpreted  as  requiring  subsystem 
stability  plus  conditions  on  the  interconnections,  usually  stipulating  a 
small  "intensity"  of  interconnection.  We  thus  obtain  a class  of  interesting 
formulations  of  the  weak  interconnection  concept. 

The  concept  of  a stable  system  formed  by  a weak  interconnection  of 
stable  subsystems  is  shown  in  Section  II  to  be  a consequence  of  the 
economic  modelling  requirements.  But  such  a system  can  arise  from 
different  considerations.  It  is  often  the  case,  as  in  a large  power -system, 
that  the  subsystem  interconnections  might  fail  thus  isolating  particular 
subsystems.  It  is  thus  desirable  that  each  subsystem  have  the  capacity 
to  operate  successfully  in  isolation  which  means  among  other  things, 
that  each  subsystem  must  be  stable. 

In  Section  IV  we  consider  an  explicit  model  for  such  systems, 
namely  differential  systems  of  the  form 

= f.(x.,t)  + h.(xp.  . . ,xN;t) 


X. 

1 


i = 1,2, ...,N  . 
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The  isolated  systems 

x.  = f.(x.,  t)  i = 1,  2, . . . , N 

i ii 

interact  through  the  additive  interconnections  h..  Starting  with  the 

assumption  of  stability  of  the  isolated  systems,  we  consider  the 

stabilizing  or  destabilizing  effects  of  the  h.'s.  The  concept  of  dissipative 

systems  of  J.  C.  Willems  plays  an  important  role  in  Section  IV  as  it 

allows  under  "weak  interconnection"  conditions  on  the  h.  the  construction 

of  a system  Lyapunov  function  from  subsystem  ones.  Hence  the  stability 

or  instability  of  a system  can  be  obtained  from  subsystem  considerations. 

The  design  of  stabilizing  controllers  for  interconnected  systems 

can  be  based  on  the  above  theory  and  is  developed  in  the  latter  part  of 

Section  IV.  In  designing  a control  scheme  for  a system  above  elements 

are  in  distinct  locations  it  cannot  be  assumed  that  system  information 

(measurements)  is  transmitted  to  a central  controller  without  a cost. 

Under  the  plausible  assumption  that  a centralized  control  scheme  is  more 

effective  than  a decentralized  one,  it  becomes  important  to  balance 

information  transmission  cost  and  control  benefit.  Such  a problem  being 

hard  even  to  express  analytically,  it  is  customary  to  consider  the 

extreme  cases  of  a centralized  versus  a decentralized  scheme.  In  particular 

Section  IV  considers  the  design  of  a decentralized  linear  feedback  control 

scheme  where  one  controller  is  associated  with  each  subsystem 

x = f (x.  t:u  ).  The  control  u.  can  be  a linear  function  of  the  i-th  sub- 
i i i’  ’ i i 

system  measurements  only,  that  being  either  the  state  x^  or  an  output 
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The  stabilization  policy  proposed  is  to  render  each  subsystem 
dissipative  enough  for  the  interconnection  terms  to  be  insufficient  to 
alter  the  stability  of  the  system.  The  important  result  of  Section  IV  is 
the  development  of  algorithmic  procedures  for  determining  both  whether 
stabilization  is  possible  and,  if  it  is,  to  generate  the  appropriate  linear 
subsystem  feedback.  This  is  done  by  showing  that  transforming  a sub- 
system into  a dissipative  one  through  linear  feedback  is  possible  if  and 
only  if  a certain  differential  game  has  a finite  value  over  closed  loop 
strategies.  The  differential  game  considered  has  linear  dynamics,  a 
quadratic  value  functional  and  is  of  infinite  duration.  An  analysis  of 
such  games,  which  are  interesting  in  their  own  right,  is  presented  in 
Section  V.  The  interesting  result  is  that  the  existence  of  a finite  value 
is  equivalent  to  a certain  algebraic  equation  (of  the  Riccati  type)  having 
a positive  definite  solution.  The  final  outcome  is  the  reduction  of  the 
stabilization  problem  to  solving  an  algebraic  equation  for  each  subsystem, 
very  much  in  the  spirit  of  the  goals  of  1.  s.  s.  t.  set  in  the  beginning  of 
this  section. 

To  complete  the  stabilization  scheme  of  Section  IV,  some 
computational  matters  are  considered  in  Section  III.  The  solution  of  a 
Riccati  equation 

A'K  + KA  + Q = KBB'K  Qj  0 

can  be  carried  out  by  an  iterative  procedure  known  as  Kleinman's 
algorithm  (though  being  a special  case  of  an  approximation  in  policy-space 
procedure  due  to  Bellman).  The  stabilization  problem  of  Section  IV  can 
be  reduced  to  a similar  equation  with  the  matrix  Q being  indefinite.  The 
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main  result  of  the  latter  part  of  Section  III  is  to  show  that  Kleinman's 
algorithm  converges  for  an  arbitrary  Q,  provided  of  course  that  a 
solution  exists.  The  proof  of  the  result  relies  on  a dynamic  programming 
approach  based  on  inequalities  of  the  Hamilton-Jacobi-Bellman  type  which 
are  treated  in  Section  III.  Some  relevant  results  on  Riccati  equations 
have  been  collected  in  Appendix  1 and  some  examples  of  an  implementation 
of  the  algorithm  on  a digital  computer  are  presented  in  Appendix  2. 

Finally,  the  dynamic  programming  approach  of  the  HJB  inequalities 
is  applied  to  the  comparison  of  optimal  systems.  Section  III  contains 
results  comparing  the  optimal  payoffs  of  two  systems  which  are 
particularly  useful  when  one  system  is  decoupled  (and  hence  easy  to 
optimize),  the  other  system  resulting  from  the  first  by  introducing 
additive  coupling  terms.  Some  estimates  of  the  performance  of  the 
optimal  decoupled  system  policy  when  applied  to  the  coupled  one  are  also 
provided. 
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II.  METZLER  STABILITY 
1.  Introduction 

In  this  section  we  intend  to  survey  and  unify  the  work  of  several 
authors  on  stability-type  questions  pertaining  to  the  modelling  of 
societal,  mainly  economic  systems.  Until  relatively  recently,  economic 
theory  was  interested  in  comparative  statics,  that  is  the  comparison  of 
possible  equilibria,  but  not  in  their  stability.  In  fact,  in  many  economic 
writings  there  was  talk  of  equilibria  without  mention  of  any  specific 
dynamical  system.  It  was  understood  that  the  relevant  dynamical 
systems  was  the  "market"  and  its  law  was  that  of  supply  and  demand. 

This  system  was  supposed  to  have  a unique,  stable  equilibrium  point 
and  if  the  equilibrium  were  to  change  because  of  exogeneous  reasons, 
the  system  would  swiftly  adjust  to  its  new  equilibrium. 

Economists  did  not  pursue  a rigorous  study  of  the  above  assumptions 
despite  the  fact  that  the  actual  economy  seldom  appeared  to  reach  its 
equilibrium.  It  was  Samuelson  [10]  who  first  pointed  out  the  need  of 
studying  the  economy  as  a dynamical  system.  This  was  done  in  the 
context  of  showing  that  a set  of  conditions  derived  on  the  basis  of  plau- 
sibility arguments  by  the  well  known  economist  Hicks  were  neither  necessary 
nor  sufficient  for  the  stability  of  dynamical  systems. 

The  reluctance  of  economic  theory  to  postulate  dynamic  models 
is  quite  justifiable.  The  adjustment  processes  of  economic  activity 
cannot  be  observed  accurately  enough  in  order  to  form  the  basis  of  a 
dynamical  theory.  Furthermore,  economic  theory  can't  be  reasonably 
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expected  to  predict  rates  of  adjustment  but  rather  the  direction  of 
adjustment:  It  can  be  predicted  that  prices  will  rise  in  response  to 
excess  demand  but  the  rate  of  price  increase  might  very  well  depend 
on  non-measurable  factors  and  lies  outside  the  domain  of  the  theory. 

This  inability  to  specify  adjustment  rates  is  an  inherent  feature  of 
economic  dynamic  models.  The  important  consequence  is  that  a model 
can  be  said  to  imply  a certain  property  only  if  that  property  holds 
independent  of  the  adjustment  rates.  A property  must  hold  in  every 
model  in  a rather  large  class  of  models  before  it  can  be  claimed  to  be 
a property  of  the  economic  system  being  modelled. 

The  goal  of  this  section  is  to  study  stability  theory  in  the  above 
context.  We  will  examine  dynamical  systems  of  the  form 

4*  = K(t)f(x,t);  K(t)  f .y(  (2.1) 

where  x is  a n-dimensional  state  vector,  f : Rn  x [t^ , 00  ) -«Rn  a 
given  function,  K(t)  a positive  definite  diagonal  matrix.  The  function  f 
incorporates  the  theoretical  specification  of  the  situation.  On  the  other 
hand  the  matrix  functions  K(t)  belong  to  a class  of  functions  3C  and 
model  the  theoretically  unspecifiable  adjustment  rates.  Of  course 
certain  restrictions  must  hold  on  f and  M so  that  solutions  to  (2.  1) 
exist  and  are  unique.  We  assume  throughout  that  such  conditions  have 
been  imposed.  A collection  of  systems  (2.  1)  is  said  to  possess  a stable 
equilibrium  point  x*  if  x*  is  a globally  asymptotically  stable 
equilibrium  point  for  every  K(t)  £ . We  shall  refer  to  such  an 

equilibrium  point  as  a Metzler-stable  one  (with  respect  to  ,X0.  Metzler 
stability  will  be  studied  for  various  classes  of  functions  f(x,  t)  and 
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adjustment  rates  ^ . Finally,  the  relation  of  the  Metzler  stability 
condition  to  recent  work  on  the  stability  of  "interconnected  systems" 
as  well  as  to  the  "aggregation"  of  large  scale  systems  will  be 
examined. 

2.  The  Hicks  Conditions  in  Linear  System  Stability 
2.  1.  The  Hicks  Conditions 

The  process  of  exchange  of  n commodities  in  a market  system 

is  accomplished  through  the  price  mechanism:  At  prices  Pp  p^, . . . , Pn 

a quantity  D.(pp  . . . , Pn)  will  be  demanded  of  the  i-th  commodity  and  a 

$ $ 

quantity  S^(Pj,  . . . , Pn)  will  be  supplied.  The  prices  Pj,  . . . , p such 
* * * * 

that  D.(pp  . . . , pn)  = S.(Pj,  . . • , Pn)  are  called  equilibrium  prices 
because  as  long  as  they  prevail  the  exchange  process  satisfies  the 
economic  needs  of  all  its  participants.  It  is  of  fundamental  importance 
to  economic  theory  to  show  that  the  equilibrium  price  is  stable,  i.  e. 
that  there  are  economic  forces  that  guide  the  prices  towards  the 
equilibrium  price. 

A special  case  of  the  above  problem  was  dealt  with  in  the  works 
of  the  early  economists  Walras,  Marshall  et  al.  They  considered  the 
case  where  all  the  prices  except  the  i-th  one  were  kept  constant  by  fiat; 
only  the  dynamics  of  the  i-th  price  p.,  were  considered.  It  is  reasonable 
to  assume  that  a price  p.  exists  for  which  demand  equals  supply  for 
the  i-th  commodity,  the  remaining  prices  being  kept  at  their  preassigned 
values.  Economic  reasoning  stipulates  that  for  p^  < p.  there  will  be 
an  excess  demand  for  the  i-th  commodity  which  will  lead  the  economic 


1 


agents  controlling  to  increase  it.  The  opposite  happens  of  p.  > p^  . 
Therefore,  if  all  prices  except  one  are  kept  fixed,  the  market  will 
lead  the  remaining  price  to  its  equilibrium  point. 

It  should  be  noted  that  the  equilibrium  price  p^  depends  on  the 
prices  Pj,  j / i,  which  have  been  kept  constant  during  the  adjustment 
of  the  i-th  market.  After  the  adjustment  is  over,  there  is  no  guarantee 
that  the  remaining  markets  will  be  in  equilibrium,  i.  e.  there  will  be  no 
excess  demand  or  supply  in  the  market  for  the  j-th  commodity  (j  / i). 
Further  economic  adjustments  will  be  needed  and  the  Walras -Marshal 
theory  can  make  no  statements  about  their  convergence  or  not  to  a price 
equilibrium. 

In  retrospect  it  seems  obvious  that  a model  which  allows  one  price 

to  adjust  while  the  others  are  kept  fixed  cannot  be  very  realistic.  A 

model  for  price  adjustment  must  consider  a process  in  which  all  prices 

adjust  simultaneously.  This  is  consistent  with  our  perception  of  the 

operation  of  the  economy.  However,  models  for  such  processes  and 

analyses  of  their  stability  did  not  appear  until  the  British  economist 

J.  Hicks  in  his  book  Value  and  Capital  attacked  this  problem  as 

follows:  The  Walras -Marshal  formulation  considers  a change  in  the 

i-th  price,  the  other  prices  being  kept  constant,  and  postulates  that  a 

price  p.  above  the  equilibrium  results  in  the  supply  exceeding  demand 

for  the  i-th  commodity  and  vice  versa.  Hicks'  formulation  is  more 

* * 

complicated.  He  assumes  that  the  equilibrium  p^,  . . • , Pn  has  been 

established  and  considers  the  effect  of  a change  in  the  i-th  price  from 

* 

its  equilibrium  position  p^  . However,  the  remaining  prices  are  not 
kept  fixed  but  are  assumed  to  adjust  so  as  to  restore  equilibrium  in 


their  respective  markets.  Specifically,  he  defines  a market  for 
n-commodities  to  be  perfectly  stable  if  the  following  conditions  hold 
[3,  p.  248]:  "A  rise  in  price  of  any  commodity  must  make  the  supply 
of  that  commodity  exceed  the  demand  (under  any  of  the  following  circum- 
stances): (a)  if  all  other  prices  are  given,  (b)  if  some  other  prices 

are  adjusted  so  as  to  preserve  equality  between  demand  and  supply  (of 
the  corresponding  commodities),  (c)  if  all  other  prices  are  so  adjusted.  " 
In  a perfectly  stable  market  an  increase  of  the  i-th  price  over  the 
equilibrium  price  p*  will  cause  excess  supply  of  the  i-th  commodity 
even  after  the  other  prices  have  adjusted.  Economic  forces  will  there- 
fore move  the  i-th  price  towards  its  equilibrium. 

Although  the  requirements  for  perfect  stability  might  be  natural 
for  an  economist,  it  is  not  clear  what  their  implications  are  for  a 
dynamic  adjustment  process.  It  is  worth  noting  that  Hicks  did  not 
postulate  his  conditions  with  reference  to  any  specific  dynamic  adjust- 
ment process.  Samuelson  [11]  studied  perfectly  stable  markets  and 
dynamic  adjustment  processes  and  showed  that  there  need  not  be  a 
relation  between  perfect  stability  and  the  convergence  of  the  adjustment 
processes  to  an  equilibrium.  To  see  this,  write  the  excess  demand 
for  the  i-th  commodity  as 

®*i^Pl»  • • • » Pn^  _ ^i^Pi>  • • • i Pn)  “ ®j(Pp  • • • » Pn)  (2.  2) 

♦ jje  £ 

From  the  definition  of  p.  , E.(pj , . . . , pn)  = 0,  i=l,...,n.  Consider 

aft 

now  the  Jacobian  of  E(|>)  at  p 

4 


P 


-li- 


lt can  be  shown  by  looking  at  small  Ap^  that  the  perfect  stability 
condition  imply  [3,  p.  315]  that  the  principal  minors  of  {a^}  must 
alternate  in  sign,  i.  e.  , 


a..  < 0 

a. . 
n 

a. . 
ij 

o 

A 

a. . 
n 

a.. 

a. . 
ij 

a. . 

n ’ 

a.. 

a. . 

7 

Ji 

JJ 

Ji 

JJ 

aki 

a.  . 
kj 

< 0 etc.  (2.  3) 


Now,  a reasonable  model  of  an  adjustment  process  is 


dP; 


IF  = ki  ^(Pp  * ' ' » Pn) 


i = 1,  . . . , n 


(2.4) 


and  in  the  case  where  E.  is  a linear  function  (and  = 1) 


E(£)  = A(£  - £ ) 
the  system  (2.4)  becomes 


d£  * 
dF  = A(£  - £ ) 


(2.5) 


(2.6) 


For  a linear  excess  demand  function  as  in  (2.  5)  it  can  be  shown  [3,  p.  315] 
that  the  perfect  stability  condition  is  equivalent  to  (2.  3)  and  hence  if 
perfect  stability  implied  convergence  of  the  adjustment  processes,  we 
would  have  that  (2.  3)  is  a sufficient  condition  for  A to  be  a stable  matrix. 
This  can  be  shown  to  be  false  through  simple  counterexamples  and  it  is 
therefore  possible  to  exhibit  perfectly  stable  markets  which  are  not 
stable  in  a dynamic  sense. 
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The  role  of  the  Hicks  conditions  to  stability  was  eventually 

clarified  by  Metzler  [6].  He  first  pointed  out  that  the  stability  of 

adjustment  processes  of  the  kind  of  (2.  4)  would  depend  strongly  on  the 

relative  speeds  of  adjustment  k^,  . . . , k^.  Second  he  noted  that  the 

verbal  statement  of  perfect  stability  by  Hicks  allows  some  markets  to 

adjust  instantaneously  while  others  do  not  adjust  at  all;  the  role  of  the 

markets  could  then  be  reversed.  Such  a stability  definition  can  only 

make  sense  if  stability  is  independent  of  the  speed  of  adjustment.  In 

this  way  Metzler  was  led  to  formulate  a version  of  what  we  call  here  a 

Metzler  stability  problem.  He  considers  a linearized  version  of  the 

adjustment  processes  around  the  equilibrium  price  p*,  and  allows 

time  invariant  but  arbitrary  positive  speeds  of  adjustment.  In  terms  of 

(2.  1)  he  considers  the  stability  problem 

(Positive  definite  ) 
diagonal  matrices  j (2.  7 

The  Hicks  conditions  are  related  to  (2.  7)  as  follows 


LEMMA  (2.8)  (Metzler)  The  Hicks  conditions  are  necessary  for 
the  stability  of  p*  for  all  systems  in  (2.  7). 


Proof.  The  characteristic  equation  of  (2.  7)  is 


c(X)  = Xn  - Cj  X11"1  + c2  Xn‘2  - . . . + (-l)n  cn  = 0 


where  cf  is  the  sum  of  all  principal  minors  of  order  r of  KA, 

H r s n.  If  now  k^  = 0 for  some  i £ {1,  2,  . . . , n}  the  characteristic 
equation  will  be 


c(X)  = X c j(X)  = 0 


where  c^(\)  is  the  characteristic  polynomial  corresponding  to  the 
matrix  (KA)11  i.  e.  , the  (n-l)x(n-l)  matrix  which  results  from 
removing  the  i-th  column  and  row  from  KA. 

It  is  necessary  for  the  stability  of  (2.  7)  for  all  K that  Cj(X.) 
has  roots  in  the  L.  H.  P.  Otherwise  we  could  choose  a positive  k. 
small  enough  so  that  some  root  of  the  characteristic  polynomial  is  in 
the  RHP.  By  repeating  the  above  argument  we  conclude  that  every 
principal  submatrix  of  A must  be  a stable  one.  Now  a necessary 
condition  for  a kx  k matrix  B to  be  stable  is  that 

(-l)k  det  B >0  . 

To  see  this  let  d(\)  be  the  characteristic  polynomial  of  B.  Clearly, 
d(M  > 0 for  some  real  positive  X..  Furthermore  d(0)  = (-1)  det  B . 

If  d(0)  < 0,  we  could  conclude  the  existence  of  a real  positive  root  of 
d(\).  Hence  we  must  have  d(0)  = (-1)  det  B^O  if  B is  to  be  stable. 
The  statement  of  the  lemma  follows  immediately.  Q.  E.  D. 

We  have  actually  proved  a stronger  condition  for  stability, 
namely 

LEMMA  (2.  9)  A necessary  condition  for  the  stability  of  p* 
for  all  systems  in  (2.  7)  is  that  all  principal  submatrices  of  A,  i.  e.  , 


etc. 


are  stable  matrices. 


It  can  be  shown  through  counterexamples  that  the  Hicks  conditions 
are  not  sufficient  for  stability  of  the  systems  in  (2.  5)  or  (2.  7).  In  an 
important  case  though  they  are  sufficient,  the  case  where  the  off-diagonal 
elements  of  the  matrix  A in  (2.  5)  are  nonnegative.  A matrix  with  this 
property  is  usually  called  an  M-matrix.  The  economic  interpretation 
of  A being  an  M-matrix  is  that  (2.  5)  refers  to  the  market  of  substitute 
commodities,  where  an  increase  in  the  price  of  a certain  commodity 
leads  to  an  increased  demand  for  all  other  commodities.  For  substitute 
commodities  and  a linear  adjustment  process,  the  Hicks  conditions  on 
A are  sufficient  for  the  stability  of  all  systems  in  (2.  5). 

PROPOSITION  (2.  10)  (Metzler)  The  Hicks  conditions  are 
necessary  and  sufficient  for  the  stability  of  all  systems  in  (2.  5)  provided 
A is  an  M-matrix. 

Proof.  Sufficiency:  The  Perron-Frobenius  theory  of  positive 
matrices  can  be  easily  applied  to  M-matrices  to  show  that  the  eigenvalue 
of  an  M-matrix  with  largest  real  part  is  real  [4,  p.  255].  Consider  now 
the  characteristic  polynomial  of  KA 

c(\)  = \n  - Cj  \n_1  + c2  \n'2  - . . . + (-1)”  cn  = 0 (2.  11) 

where  c^  is  the  sum  of  all  principal  minors  or  order  r.  If  the  Hicks 
conditions  hold,  cr  * ( - 1 ) >0  and  thus  all  coefficients  of  c(\)  are 

positive.  This  shows  that  (2.  11)  has  no  roots  on  the  positive  real  axis 
and,  by  the  result  stated  in  the  beginning  of  the  proof,  it  can  have  no 
roots  with  nonnegative  real  parts.  The  necessity  part  is  Lemma  (2.  8). 


Q.E.  D. 
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The  obvious  question  is  to  establish  necessary  and  sufficient 
conditions  for  the  stability  of  all  systems  in  (2.  4).  The  necessary 
condition  in  Lemma  (2.  9)  is  much  stronger  than  the  Hicks  conditions, 
however  it  is  not  clear  whether  it  is  also  sufficient. 


2.  2 Generalizations  of  the  Metzler  Stability  Problem 

We  now  want  to  extend  the  problem  posed  by  Metzler  by  allowing 
time  varying  speeds  of  adjustment.  Namely,  we  consider  the  systems 


— P = K(t)  A(p-p*) 

K(t)  6 - {K(t):  Diagonal  matrices,  continuous 

in  t and  0<ks  k.  (t)  } 


(2.  12) 


and  seek  conditions  on  A for  p*  to  be  a stable  equilibrium  point  for 
every  system  in  (2.  12).  We  provide  next  a sufficient  condition  for  this 
to  happen 


PROPOSITION 


(2.  13) 

Define 

A by 

a. . 

for 

i = j 

U 

| a. . | 

for 

i/j 

1 ir 

If  A is  a stable  matrix,  all  systems  in  (2.  12)  are  stable. 


Proof.  By  a theorem  of  Ostrowski  [8]  there  exists  a positive 
diagonal  matrix  D such  that  DAD" 1 is  row  diagonally  dominant,  i.  e. 


a.. 

n 


+ 


L 

j/i 


d.  s -n  < 0 
i 


(2.  14) 


for  some  7T  and  all  i = 1,  2, , n.  Consider  now  the  system 

x = K(t)Ax.  The  stability  of  the  origin  for  this  system  implies  the 
stability  of  p*  in  (2.  12).  We  will  proceed  to  show  that  x = K(t)Ax  is 
stable.  Let  y^(t)  = d..x.(t)  and 

y(t)  = max  ( | y j (t)  | , |y2(t)|,...,  |yn(t)|) 

Furthermore  define  the  index  set  J(t)  as 

J(t)  = {j  |yj(t)|  = y(t)3 

The  function  y(t)  is  continuous  but  not  necessarily  differentiable  for 
every  t.  We  will  show  that  the  right-hand  derivative  of  y(t)  exists 
and  satisfies  a differential  inequality  from  which  we  will  derive  the 
required  stability  of  x = K(t)Ax. 

From  the  definition  of  y(t)  it  follows  that  y(t)  = 0 is  equivalent 
to  x.(t)  = 0.  We  will  consider  therefore  the  right-hand  derivative  of 
y(t)  whenever  y(t)  / 0.  Consider  the  expression 


D(it)  = I(t±ASL^l£)  ; 


At  > 0 


> 


y(t)  /o 


Since  y(t)  / 0 it  follows  that  the  | y^(t)  | j € J(t)  are  differentiable 
at  t.  Furthermore  if  d/dt  | y^(t)  | > d/dt  |y^(t)|  for  k,  i f J (t), 

| yk( T ) I 2 |yf(T)|  for  T ^ [t , t+  A } and  some  A.  Therefore,  if 
d/dt  j yk(t)  ( = {d/dt  (y^(t)|]  there  is  a A ^ such  that 

| y"k^T ^ I s | y j ( T ) | for  j £ J{t)  and  tp  [t,  t+Aj].  Furthermore  the 
continuity  of  |y^(t)j  implies  that  since  |y^(t)|  < |y^(t)|  = y (t)  for 
j f J(t),  i / J(t),  the  same  inequality  holds  for  a neighborhood  of  t, 


namely 


ly^r)! 


for  T £ [t,  t+A?]  . 


For  0 s Ats  min  [a^,  A^]  the  expression  D(At)  becomes 


D(  At)  = 


y(t+ At)  - y(t) 


yk(t+At)|  - |yk(t)| 


where 


k - arg  max^j^^d/dt  J yj(t)  J } and  hence 


4t"m0+  DtM)  ' 5f  bk(t)l 


We  now  have  following  bounds  for  the  right-hand  derivative  (which  we 
denote  by  RHD) 


-1 


It  follows  from  [14]  that  exp( -kJTt)*  y(t)  is  decreasing  in  t and  thus 
y(0)  2 exp(-k7Tt)-  y(t)  or  y(0)  exp(k7Tt)  ^ y(t)  as  was  claimed. 

The  above  proves  that  y(t)  -*  0 and  so  does  x(t).  The  stability  of 
the  system  x = K(t)Ax  has  been  proved.  Q.  E.  D. 

Remark.  The  Metzler-Hicks  stability  conditions  of  Lemma  2.  8 
are  a special  case  of  those  in  Proposition  2.  13.  Similar  conditions  appear 
in  [4],  [12],  [15]  but  not  in  the  context  of  Metzler  stability. 


Generalizing  a step  further  we  might  consider  the  system  matrix 
A in  (2. 12)  to  be  time  varying.  What  conditions  on  A(t)  will  guarantee 
stability  of  p*?  It  can  be  seen  by  counter-example  that  it  is  not  sufficient 
to  ask  that  the  conditions  of  Proposition  2.  13  hold  at  each  time  instant  t, 
i.  e.  A(t)  be  a stable  matrix  for  every  t.  However,  it  is  known  [13],  [7] 
in  the  context  of  the  study  of  time  varying  linear  systems  x = A(t)x,  that 
there  is  a class  of  matrices  A(t)  for  which  stability  of  A(t)  for  every 
t implies  the  stability  of  the  time  varying  system  x = A(t)x.  Namely, 
consider  the  system 


x = ,A(t)x 


(2. 


where  the  A(t)  matrix  is  of  the  form 


-a. 

l 


for  i = j 


a (t)  = 
ij 


(2.1 


a.  .•  e..(t) 
ij  ij 


for  i / j 


and  e..(t)  are  continuous  functions  such  that  |e..(t)|  £ 1.  In  the  above 
ij  ' ij  1 


references  [12],  [7]  it  is  shown  that  (2.15)  is  globally  stable  provided  the 


matrix 
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(2.  17) 


is  a stable  M-matrix.  The  stability  of  the  systems  in  (2.15)  has  been 
called  connective  stability  in  [13],  It  can  be  justified  if  we  consider  the 
off-diagonal  terms  of  A(t)  as  interconnection  coefficients  which  can 
vary  in  an  arbitrary  manner  due  to  the  presence  of  the  e^(t)  coefficients 
in  (2.  16):  Connective  stability  means  that  the  stability  of  (2.  15)  is 
independent  of  the  strength  of  the  interconnection  terms. 

We  can  combine  the  notion  of  connective  stability  with  Metzler 
stability,  i.  e.  stability  of  all  systems  in  (2.12)  by  noting  that  the  stability 
condition  (2.  5)  guarantees  the  stability  of  all  the  systems  in  (2.  12).  We 
have  thus 


PROPOSITION  2.18.  Let  A(t)  be  defined  as  in  (2.16)  and  let 
A in  (2.17)  be  a stable  M-matrix.  The  equilibrium  p*  of  the  system 

A.  p = K(t)  A(t)  (p  - p*)  (2.19) 

is  globally  asymptotically  stable  for  all  K(t)  £ ^ as  in  (2.  12). 

Proof.  As  in  the  proof  of  Proposition  2. 13,  the  fact  that  A is  a 
stable  M-matrix  implies  the  existence  of  a positive  diagonal  matrix  D 
such  that  DAD  * is  row  diagonal  dominant.  Again  let  us  consider  the 
system 

x = K(t)  A(t)  x 


with  the  understanding  that  the  stability  of  the  origin  is  equivalent  to  the 


stability  of  p*  in  (2.  19).  Let  now  y^  = d^x.  and  y(t)  = max(|y 
Then  there  exists  an  index  set  J(t)  such  that  J(t)  = {j|y  (t)|  = 


in  the  proof  of  Proposition  2.13  we  can  show  that  the  right  hand  derivative 


RHD(y(t))  exists  and  RHD  y(t)  = max 
We  have,  for  y(t)  / 0 


RHD  y(t)  = max  sgn  y.(t) 

j€J(t)  J 


£ max 
j€J(t) 


As  in  the  proof  of  Proposition  2. 13  we  can  conclude  from  RHD  y(t)  s k7Ty(t) 


y(t)  s exp(k7Tt)  y(0) 


which  shows  that  y(t)  -*  0 and  completes  the  stability  proof 


Remark.  The  proofs  of  connective  stability  in  [12],  [13]  and  [7] 


employ  the  Lyapunov  functions 


The  stability  property  of  the  matrix  A implies  that  the  d.  can  be  chosen 
in  such  a way  that  the  derivative  of  v(x)  is  everywhere  negative.  A 
consistent  choice  of  d^  is  not  possible  in  case  the  speeds  of  adjustment 
k^(t)  are  time  varying.  This  difficulty  can  be  avoided  by  using  the 
Lyapunov  function  of  the  previous  proofs. 

3.  Nonlinear  Metzler  Stability  Problems 

3.  1 A Lyapunov  Function  Approach 
Consider  the  collection  of  systems 

x(t)  = K(t)f(x,  t)  ; K(t)€^ 

The  origin  is  assumed  to  be  an  equilibrium  point,  i.  e.  f (0 , t)  = 0 for  all  t. 

The  Metzler  stability  problem  is  to  establish  conditions  on  f(x,  t)  which 

guarantee  that  the  origin  is  globally,  asymptotically  stable  for  all 

K(t)  £ .'/{y  A natural  way  to  attack  this  problem  is  through  Lyapunov  functions 

Consider  a scalar  function  v(x,  t)  with  continuous  and  such  that 

ox  ’ dt 

(i)  0j(  ||x|| ) £ v(x,  t)  £ 02(||x||  ) 

where  0i(  |j  x || ) are  nonnegative,  nondecreasing  functions  such  that 
0.(0)  = 0,  li^n  ||  x ||  _ oo  0(|lxl|)  = 00  • Assume  that  it  can  be  shown  that  for 
every  K(t)  € the  time  derivative  of  v(x,t)  along  the  corresponding 
trajectory  of  (2.1)  satisfies 

(ii)  ~ v(x(t),t)  £-03(||x||) 


where  03(||x||)  satisfies  the  same  conditions  as  0j,  0^.  Then  the  global 
asymptotic  stability  of  the  origin  for  every  K(t)  £.)#.  follows  by  a standard 


-23- 


Lyapunov  stability  theorem  [2,  pg.  458].  In  addition  to  the  problem  of 
selecting  a proper  v(x,  t),  the  checking  of  the  derivative  condition  (ii) 
is  extremely  cumbersome. 

A simple  condition  for  (ii)  to  hold  would  be  as  follows.  Consider 
for  simplicity  time-invariant  Lyapunov  function  v(x)  € C*(Rn)  and  time 
invariant  f functions.  The  derivative  in  (ii)  can  be  written  as 


dv(x) 

dt 


dv 

dx. 

l 


L(x)  k. 


(2.  20) 


Unless  each  term  0v/dxj  f.(x)  in  (2.  20)  is  nonpositive  and  at  least  one 
is  negative,  there  will  be  a choice  of  k.  which  will  make  dv/dt  positive 
(provided  there  is  no  upper  bound  in  the  k^).  In  any  case,  the  condition 


dv 


dx.  i 
i 


f . (x)  s;  0 


i € [1, 2, . . . ,n] 


dv 

dx.  ij 
J 


f.(x)  < 0 for  some  j € [1, 2,  . . . ,n] 


(2.  21) 


is  sufficient  for  the  stability  of  (2.  1),  coupled  with  some  technical 
conditions  as  in  the  following: 

LEMMA  2.  22:  Consider  the  class  of  systems 

^x  = k(t)  f(x)  ; K(t)  (2.23) 

and  assume  the  existence  of  a C (R  ) positive  definite  function  v(x) 


satisfying 
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ir  fiWs0 


i € [1,  2 n] 


tr  Vx)<  0 

j 


for  some  j 6 [1,  2,  ...  ,n] 


Then  all  the  systems  in  (2.  23)  are  globally  asymptotically  stable. 


Proof.  Note  that 


n 


av  _ y 
dt  " ^ 


i dx. 
1=1  l 


f.(x)  ^ 0 


which  satisfies  the  assumptions  of  the  global  as.  stab,  theorems  of  [16]. 

Q.  E.  D. 


Although  Lemma  2.22  is  extremely  simple  it  has  interesting 
consequences.  Let  us  apply  it  to  linear  systems  of  the  form 


x = K(t)  Ax 


(2.  24) 


1 


I 


I 

I 


with  K(t)  6 •'*( j as  in  (2.  23).  A quadratic  Lyapunov  function  v(x)  = x'Mx, 
M >0  would  satisfy  the  conditions  of  Lemma  2.  22  if,  for  all  x 

f-  = (e.'Mx  + x'Me.)(Ax).  = x'M  .A.  x s 0 
ox.  1 1 11  .11. 

where  e.:  i-th  unit  vector,  M ^i-th  column,  A.:i-th  row.  This  can  only 
happen  if  the  i-th  row  of  A is  proportional  to  the  i-th  column  of  M. 

These  conditions  can  be  satisfied  if  A is  a symmetric  negative  definite 
matrix.  Then  v(x)  = -x'Ax  is  a positive  function  and 
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4^-  = -x'AKAx  - x'AKAx  = 2x'AK(t)Ax<  -2kx'A'Ax 
at 


is  negative  everywhere,  providing  thus  a direct  verification  of  the 
conclusion  of  Lemma  2.  22. 

It  should  be  noted  that  the  conditions  of  Lemma  2.  22  are  far  from 
necessary  for  the  stability  of  all  systems  in  (2.23):  In  the  linear  case, 
the  only  systems  satisfying  these  conditions  correspond  essentially  to 
symmetric  negative  definite  matrices.  However,  all  A-matrices 
satisfying  the  conditions  of  Proposition  2.13  are  Metzler  stable,  and 
these  A's  are  not  covered  by  Lemma  2.  22.  Thus  the  condition 
3v/dx.  f.  ^ 0 for  all  i can  fail  at  some  x without  implying 
instability.  On  the  other  hand,  if  dv/dx.  f.  > 0 for  some  i and  all  x 
in  a neighborhood  Nq  of  the  origin,  the  system  can't  be  stable  as  dv/dt 
can  be  made  positive  on  Nq.  Instability  follows  then  by  standard 
theorems  [16].  The  above  remarks  provide  a weak  converse  to  Lemma  2.  22. 


3.  2 The  Malishewski  Conditions  for  Metzler  Stability 
A class  of  sufficient  conditions  for  Metzler  stability  of  nonlinear 


systems  were  presented  by  Malishewski  [5].  In  this  section  we  examine 
his  conditions  and  show  that  they  follow  in  large  part  from  Lyapunov 
function  considerations,  similar  to  those  that  lead  to  Lemma  2.  22.  First 
we  present  a brief  summary  of  the  framework  of  Malishewski's  work. 

Malishewski  introduces  the  notion  of  goal-oriented  elements 
operating  jointly.  Each  "element"  is  a scalar  controller  and  the  i-th 
control  is  a scalar  c.,  i = 1,2,  ...  , n.  The  joint  application  of  the  control 


! 
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1 


values  c^,  i = 1,  2,  . . . ,n  causes  each  controller  to  receive  a "mis- 
match" message,  which  depends  on  the  controls  c^,  i = l,...,n 
applied.  The  mismatch  message  of  the  i-th  element  is  determined  by 
a deterministic  function  6.(c, , c_,  . . . , c ) which  is  referred  to  as  an 
indicator  function.  The  goal  of  each  element  is  to  receive  a zero  mis- 
match message  and  therefore  the  joint  operation  of  the  n elements  will 
be  successful  if  each  element  applies  a control  of  magnitude  c*  such  that 
6.(c^,  c*,  . . . c*)  =0  i = 1,  2,  . . , n.  Two  questions  arise  in  this  setup. 
First,  the  purely  algebraic  question  of  the  existence  of  the  equilibrium 
values  c*,  i = 1,  2,  . . . ,n.  Second  the  question  of  whether  the  rules  of 
operation  of  the  controllers  will  lead  to  convergence  of  the  controls  to 
the  equilibrium.  Concerning  the  rules  of  operation,  each  element  is 
assumed  to  know  nothing  about  the  functional  form  of  the  indicator 
functions  and  thus  the  possibility  of  explicit  calculation  of  c*  by  the 
controllers  is  excluded.  The  control  strategies  can  be  based  on  the 
knowledge  of  the  controller's  mismatch  value  only.  Malishewski 
examines  two  kinds  of  control  strategies.  First  the  strategy 

= !.(c  jj  . • . , cn)  i=l,2,...,n  (2.25) 

under  which  the  problem  is  a standard  asymptotic  stability  problem. 

More  interesting  is  the  class  of  strategies  described  by 

sgn  c.  = sgn  6.(c1}  c2, . . . ,cn)  i=l,2,  ...,n  (2.26) 

where  sgn(  ) is  the  well  known  sign  function.  Obviously  the  stability  of 
(2.  26)  is  a Metzler  stability  problem  as  defined  earlier  in  this  section. 
Note  that  the  trajectories  in  (2.  26)  must  satisfy  a nondegeneracy 


L 


condition,  that  prohibits  the  existence  of  a time  sequence  { tnJ  such  that 
c.(tn) -»  0 but  6j(c(tn))/*0.  The  degeneracy  condition  guards  against 
convergence  to  nonequilibrium  points.  In  the  same  spirit,  a lower  bound 
was  imposed  on  the  adjustment  rates  of  the  Metzler  stability  problems  we 
encountered  earlier  in  thi  section. 

A final  remark  should  be  made  about  Malishewski's  model  before  we 
proceed  to  the  convergence  conditions.  It  might  be  realistic  to  impose  hard 
constraints  on  the  control  magnitude  of  the  form 

a.  ^ c.  ^ b. 
ill 

These  constraints  can  be  incompatible  with  the  controller  actions  in  (2.  25) 
and  (2.  26)  as  in  the  case  where  6.(cj, . . . , b.,  . . . , cn)  > 0,  and  thus  the 
control  policy  leads  to  the  violation  of  the  magnitude  constraint.  This 
difficulty  can  be  avoided  by  modifying  the  indicator  functions  as 

6^(c  i > • • • > cn^ 


if  a.  < c.  < b. 

ill 


6i(cl’  ‘ * ’ , cn)  = 


if  c.  = a.,  6.  < 0 


c.  = b.,  6.  > 0 

i i’  i 


Henceforth  we  will  assume  that  such  modification  has  been  carried  out  on  the 

indicator  functions  6.. 

i 

Malishewski  considers  continuous  indicator  functions  which  satisfy  a 
class  of  rather  formidable  looking  conditions  and  examines  when  these 
conditions  imply  Metzler  or  ordinary  stability.  We  will  show  that  the 
Metzler  stability  results  can  all  be  viewed  as  following  from  Lyapunov  type 
considerations  as  in  Lemma  2.  22.  Let  us  first  review  the  conditions 
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imposed  on  the  function  indicators. 

Let  c=  (cj,...,cn)  and  c + Ac  = (cj  + ACj, . . . , cn  + AcJ  be 
arbitrary  control  vectors  and  let 

A6^(c)  = 5 .(£  + A£)  - b^(c)  i = 1, . . • , n 

The  indicator  functions  satisfy  Condition  I if  the  following  hold:  Let 
= { i [ Ac.  > Oj  , 

I,  = (i|Ac.  < Oj  , 

I_  = [i  | Ac.  = 0}  . 

For  any  £,  A£  we  must  have 

L A6.(£)  - E A6.(£)+  E I A 6 (£)  I < 0 

i€l>  1 i€I<  i€I_ 

Condition  II  is  satisfied  if  the  following  holds:  Let 
We  must  have 

A &k(£)  • Ack<  0 (Ac  4 0) 

Condition  111  holds  if 
n 

E A6.(c)  Ac.  < 0 ( A£  4 0) 

i=l  1 1 

It  should  be  noted  that  there  is  no  implication  relation  among  the  three 
conditions.  In  fact  it  will  be  shown  that  they  arise  from  essentially  different 
considerations.  Several  ways  of  interpreting  them  are  possible  but  the  most 


(Ac  4 0) 

| Ack|  = max.  | Ac.  | . 


enlightening  comment,  which  applies  to  all  three,  is  that  each  condition 
guarantees  that  the  i-th  indicator  function  is  most  sensitive  to  the  i-th 
control.  Thus,  a controller  applying  a large  positive  change  in  control 
is  guaranteed  a decrease  in  his  mismatch  message,  regardless  of  the 
actions  of  the  other  controllers. 

A more  transparent  representation  of  the  three  conditions  is  possible 
in  case  the  indicator  functions  are  continuously  differentiable.  By  taking 
limits  as  A£  -*  0 in  each  condition  we  see  that  each  implies  a corresponding 
condition  in  differential  form,  namely 


Condition  I. 


a &(£)  ^ 

+ 2-, 

5Ci  jA 


dSJc) 


i = 1,  . . . ,n 


Condition  II. 


d6.(c) 


+ £ 

i/i 


d6.(c) 


i 1,..., n 


Condition  IIIT 


I S6.(c)  96.(c)  \ 1 

\ dcj  + dci  / 


s 0 (all  minors) 


These  conditions  must  hold  at  every  £ vector.  If  the  differential 
conditions  hold  everywhere  as  strict  inequalities,  they  can  be  integrated 
to  yield  the  corresponding  difference  conditions. 

The  differential  conditions  above  can  be  viewed  as  generalizations 
to  nonlinear  systems  of  well  known  stability  conditions  for  linear  systems. 
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Conditions  1^,  H^  state  that  the  matrix  (d6(c)/d£)  is  column  and  row 
dominant  respectively  and  we  can  conclude  through  Gersgorin's  theorem 
that  the  eigenvalues  of  (d6(c:)/Bc)  are  in  the  L.  H.  P.  for  every  £. 
Condition  IH^  states  that  (d6(£)/d£)  is  negative  definite  and  thus  its 
eigenvalues  are  also  in  the  L.  H.  P.  Furthermore  1^,  II  imply  the 
conditions  of  Proposition  2.  13  which  guarantee  Metzler  stability  for 
linear  systems.  However,  the  above  relations  are  of  value  mostly  as 
mnemonics,  as  there  is  no  fundamental  relation  between  Gersgorin's 
theorem  and  the  stability  that  conditions  1^,  IID  entail.  As  hinted  before 
the  meaning  of  Maleshewski's  conditions  is  best  clarified  through 
Lyapunov  function  type  arguments  that  led  to  Lemma  2.  22.  Namely,  it 
can  be  easily  seen  that  there  exist  positive  scalar  functions  which  are 
monotonically  decreasing  for  every  trajectory  of  the  class  (2.26),  provided 
the  indicator  functions  satisfy  Conditions  I or  II. 

For  indicator  functions  satisfying  Condition  I consider  the  positive 
scalar  function 

v(£)  = £ | Me)  | 
i=l  1 

We  will  show  that  v(c(t))  is  decreasing  in  time  for  every  trajectory  of  the 
class  (2.  26).  Since  v(£)  is  not  everywhere  differentiable  with  respect  to 
£,  we  cannot  conclude  that  v(c(t))  is  differentiable  in  t.  We  consider 
instead  a derivate  [14,  p.  97]  of  v(c(t)),  namely 

. v(c(t  + At))  - v(c(t)) 

D v(c(t))  = lim 

At-«0+  At 


r 


| 


We  will  show  that  D+  v(c(t))  £ 0.  In  fact, 
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n |6.(c(t+ At))|  - | 6.(c (t) ) | 

D+v(c(t))  = lim  £ 

At-*  0 i=l 


(2.  27) 


At 


n 6.(c(t+At))  sgn  6.(c(t+At))  - 6.(c(t))  sgn  6.(c(t)) 
tt—  V"  i _ 1 l i 


lim  

At-*0T  i=l 


At 


Let  us  assume  that  fi^ctt))  i 0 for  all  i.  Then  for  small  enough  At, 


we  have 


(2.  28) 


and 


(2.  29) 


sgn  6. (c(t+ At))  = sgn  6.(c(t)) 

sgn  [c i ( t-f- At)  - c.(t)]  = sgn  [k^t  * e^ctt))] 

= sgn  fi.(c(t)) 

Putting  (2.  27),  (2.  28),  (2.  29)  together  we  get 
, n 

D+v(c(t))  = lim  — £ sgn  [c.(t+At)  - c.(t)]  • A 6.(c(t)) 

At-0+  At  i=l  1 


Condition  I implies  that  the  expression  of  which  the  lim  sup  is  taken  is 
negative  for  At  small,  and  thus  D+v(c(t))  =£  0 for  all  t. 

We  assumed  above  that  6.(c(t))  / 0 for  all  i.  It  can  be  shown  that 
this  assumption  is  not  necessary  to  show  that  D+v(c(t))  $0.  In  fact,  let 
6.(c(t))  = 0.  Then 

D+v(c(t))  = lim  4*  ( H A6.  • sgn  Ac  + | 6 .(c(t+ At)  | J 
At-0  + \ i/j  J / 


1 

*.  * 


u 


Note  that  Condition  I stipulates  that 


Zj  A6.  sgn  Ac.  + |a6.|  < 0 

Ac./O  1 1 Ac.=0 

i J 

and  also  that  m(t+At)  - cj(t)  =“  0(At)  while  c^t+At)  - c.(t)  ^ 0(At).  A 
continuity  argument  can  then  be  used  to  show  that  the  complete  form  of 
Condition  I implies  that  D+v(c(t))  s 0. 

Using  now  a real  analysis  result  from  [14,  p.  98]  we  conclude  that 
v(c(t))  is  decreasing  in  t and  is  differentiable  almost  everywhere. 

A similar  monotonicity  result  can  be  given  under  Condition  II.  Let 
us  assume  that  an  equilibrium  point  c*  exists.  Consider  then  the  scalar 
function 

v(c)  = max  |c.  - c*| 
i 

and  consider  the  derivate  of  the  function  v(cjt))  along  a trajectory 
satisfying  (2.  26).  Using  the  same  arguments  as  in  the  proof  of  Proposition 
2.13  we  can  conclude  that  the  right  hand  derivative  of  v(c(t))  exists 
everywhere  and 

RHD  [v(c(t»]  = max  A | c (t)  - c*| 

j€J(t)  dt  3 3 


where 

J(t)  = j|  |c  (t)-c*|  = max  |c  (t)-c*|j 

( I J 3 i€{l, . . . n}  1 1 

Note  that  for  v(£(t))  / 0 

RHD  [v(c(t»]  = max  ~ |c.(t)  - c*| 
j€J(t)  dt  3 3 

= max  sgn  (c.(t)  - cf)  A (c  (t)  - cf) 
j€J(t)  3 3 d 3 3 

= max  sgn  (c.  (t)  - cp  (6-(£(t))  - 6d£*))k. 

Ix-  T / \ J J J J J 


Condition  II  implies  that  the  last  expression  is  negative  for  £(t)  / c*  and 
hence  RHD[v(c(t))]  s 0.  From  the  result  in  [14,  p.  98]  quoted  above  we 
can  conclude  that  v(c(t))  is  decreasing  in  t,  and  differentiable  almost 
everywhere. 

The  scalar  function  v(£)  = max.  | c^  “ c?|  is  acceptable  as  a 
Lyapunov  function  and  using  standard  Lyapunov  type  theorems  it  follows 
that 

PROPOSITION  2.  30  (Malishewski).  Suppose  that  an  equilibrium 
point  £*  exists  and  Condition  II  is  satisfied.  There  the  equilibrium 
point  c*  is  globally  asymptotically  stable  for  any  system  in  (2.  26). 

In  contrast  to  vie)  = max.  I c.  - c*|  , the  scalar  function 
v(c)  = 1 cannot  be  used  to  conclude  stability  unless  it  can  be 

shown  to  satisfy  extra  conditions:  The  strongest  result  for  systems  in 
(2.  26)  is  a convergence  result  for  bounded  trajectories. 

PROPOSITION  2.  31  (Malishewski).  Let  Condition  I hold.  Then 
each  bounded  trajectory  of  the  systems  in  (2.  26)  converges  to  the 
equilibrium  point. 

In  the  monotonicity  proofs  above,  the  hypotheses  i.  e.  , Conditions  I, 
II,  were  not  used  in  their  full  strength.  In  the  Condition  II  proof,  it  was 
only  required  that  the  inequality  Ac^  ' < 0 hold  at  c*  only,  while 

Condition  II  requires  the  inequality  to  hold  for  every  £.  In  the  Condition  I 
convergence  proofs  it  is  only  necessary  that  the  relevant  inequality  hold 
locally,  i.  e.  for  £,  £ + A£  with  A£  in  some  neighborhood  of  0,  while 
Condition  I requires  the  inequality  to  hold  for  all  £ and  A££  Rn.  It  turns 
out  that  this  strengthened  form  of  I,  II  guarantees  an  important  property 
of  these  models,  i.  e.  existence  and  uniqueness  of  £*,  such  that  M£*)  = 0, 
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thus  providing  a justification  for  alluding  to  _c*  as  the  equilibrium  point 
in  Propositions  2.  30  and  2.  31.  Furthermore,  the  strong  form  of 
Condition  I guarantees  that  c*  is  the  only  point  where  dv/dt  = 0,  a fact 
that  is  important  if  special  type  stability  theorems  are  to  be  used  which 
contain  statements  about  dynamic  trajectories  converging  to  the  set 
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The  stability  results  that  Malishewski  obtains  under  Condition 
III  concern  the  uniqueness  and  existence  of  an  equilibrium  point  c*  as 
well  as  the  stability  of  trajectories  satisfying  (2.  25),  i.  e.  the  ordinary 
differential  equations  c.  = 6.(£)  i = 1,.  ..  ,n.  No  results  about 
trajectories  satisfying  (2.  26)  i.  e.  , sgn  = sgn  6^),  appear.  This 
unsymmetry  can  be  clarified  in  terms  of  Lyapunov  function  arguments. 
The  natural  Lyapunov  functions  for  systems  satisfying  condition  III  are 

vl<£)  = Z 6f(£) 

i=l 

j v2(£)  = Z (Cj  - cf)2 


and  it  can  be  easily  seen  that  both  are  monotonicallv  decreasing  along 
trajectories  satisfying  (2.  25).  However,  when  dealing  with  Metzler-type 
stability  as  in  (2.  26)  one  must  look  at  stronger  conditions  like  the  conditions 
of  Lemma  2.  22,  namely 


d v 
dc. 


6i(c)  £ 0 


Computing  the  above  expressions  for  Vj,  v?  and  assuming  differentiable 


1 
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indicator  functions  we  get 


Sv.  n 96. 

5T  6i(^  = 2 .£  6j(-  9^  • *i(^ 


£ 0 


(2.32) 


5?  6i<c»  = 2 «=i  - cf>  si(c) 
1 


£ o 


(2.  33) 


Neither  inequality  is  implied  by  Condition  III.  However,  when  trajectories 
of  (2.  25)  are  considered  the  total  derivatives  become 


dv,(£) 

— i — = 2 E 6.  6.  SO 
dt  M 1 J 


dv2(£) 

“dt 


n 


= 2 £ (c.  - cf)  (6  (c)  - 6.(c*))  s 0 

i=  1 1 1 1 1 


if  Conditions  HI^,  III  hold.  Thus  Lyapunov  theorems  based  on  Vp  v^ 
can  be  proven  for  (2.  25)  type  trajectories  but  not  those  in  (2.  26). 

We  saw  above  that  Conditions  I,  II  entail  very  desirable  stability 
properties,  much  more  desirable  than  those  of  Condition  III.  However, 
there  is  a rather  important  drawback  in  their  formulations:  They  are 
not  invariant  with  respect  even  to  a change  of  scale:  In  an  economic,  say, 
model  let  us  assume  that  either  Condition  I or  II  is  satisfied.  Consider 
now  the  possibility  of  another  observer  expressing  the  same  situation  but 
in  terms  of  the  variables 


c^t)  = Jii  c.(t) 


i = 1, 2,  . . . , n 
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with  ^ positive  time  invariant  scalars.  The  model  in  terms  of  the  c" 
will  have  the  same  stability  properties  as  the  one  expressed  in  terms  of 
c.,  but  it  will  not  necessarily  satisfy  neither  condition  I nor  II.  This  can 

> 

be  seen  most  easily  by  examining  the  differential  conditions  and  observing 
that  diagonal  dominance  properties  are  not  necessarily  preserved  under 
the  operation  K(*)K  where  K is  a positive  diagonal  matrix.  On  the 
other  hand  Condition  III  is  preserved  under  such  transformations.  The 
conclusion  is  that  if  a certain  economic  model  involving  indicator  function 
has  been  constructed  and  one  wants  to  test  it  for  Metzler  type  stability, 
testing  for  Conditions  I,  II  will  most  probably  give  negative  results  even 
though  the  model  "essentially"  satisfies  those  conditions.  The  reason  is 
that  one  can  never  be  sure  of  choosing  the  "right"  scale  and  most  likely 
he  won't. 

* 

A simple  way  to  get  around  this  difficulty  is  to  modify  Conditions  I, 

•i 

II,  as  follows:  Condition  Ij^(IIj^)  is  satisfied  if  there  exists  a positive 

scale  transformation  which  transforms  the  indicator  functions  into  ones  > 

for  which  Condition  I(II)  is  satisfied.  Unfortunately  there  doesn't  seem 
to  exist  an  effective  way  of  verifying  these  modified  conditions  which  are, 

therefore,  of  little  value.  It  is  interesting  to  note  though  that  if  bb/bc 

' 1 2 

is  diagonally  dominant,  the  matrix  K {bj>/bc)  K~  will  be  a stable  M- 


■ ■■  II 


-37- 

The  natural  conjecture  that  Metzler  stability  is  implied  by  the  condition 
that  {S_£>/6  £}  is  a stable  M-matrix  can  be  easily  shown  to  be  false. 

What  is  true  is  the  following  generalization  of  the  Metzler -Hicks  conditions 
to  nonlinear  systems 

PROPOSITION  2.  34  (Malishewski).  Suppose  that  d6jc)/dc 

is  a stable  M-matrix  for  all  £ and  suppose  that  there  exists  vectors 

1 2 1 2 
£ < £ such  that  £>.(c  ) 2 0,  6^(c  ) s 0,  i = 1,  2, . . . , n.  Let 

1 2, 

S = [c|c  ^cs  c J . Then  there  exists  a unique  equilibrium  c*  in  S and 
each  trajectory  satisfying  (2.  26)  that  starts  in  S,  i.  e.  c(0)  €S,  converges 
to  c*. 

The  proof  of  this  proposition  relies  heavily  in  arguments  involving 
"reflecting  barrier  boxes.  " It  is  not  directly  related  to  Lyapunov  function 
type  proofs,  and  can't  be  easily  explained  by  the  arguments  used 
to  explain  Conditions  I,  II.  The  hypotheses  of  Proposition  2.  34  are 
interesting  in  that  they  both  generalize  the  Hicks  conditions  and  are 
invariant  under  scale  transformation.  In  that  sense  they  are  a satisfactory 
generalization  of  the  results  that  were  obtained  by  Metzler  in  the  linear 
system  case. 

4.  The  Hicks  Conditions  in  the  Analysis  of  Large  Systems 

The  goal  of  large  scale  system  theory  is  to  identify  systems  of  high 
dimensionality,  the  properties  of  which  can  be  efficiently  determined  by 
special  methods  of  the  theory.  For  example,  the  decomposition  theory  of 
linear  programming  identifies  a class  of  linear  programming  problems 
which  can  be  efficiently  solved  by  the  Dantzig- Wolfe  Decomposition 


Algorithm.  The  Hicks  conditions  play  an  interesting  role  in  large  scale 
system  theory  by  establishing  a class  of  systems  about  which  one  can 
efficiently  answer  stability  questions  [2],  [12],  [7],  [15].  We  will  survey 
some  typical  results  and  we  will  provide  extensions  to  Metzler  stability. 

Suppose  we  are  interested  in  the  stability  of  the  high  dimensional 
system 

x = f(x,  t)  (2.  35) 

Assume  that  the  system  possesses  unique  solutions  for  every  initial 

condition  and  that  the  origin  is  the  unique  equilibrium  point.  The  following 

structure  is  postulated  on  system  (2.  35):  It  is  "composed"  of  S inter- 

n. 

connected  dynamic  subsystems;  the  i-th  of  which  has  a state  x.  € R *. 

The  "total"  system  state  is  x = (x^,  . . . , xg).  The  system  equation  (2.  35) 
is  expressible  as 

xi  = g.(x.,t)  + h.(t,x)  i = 1 S (2.36) 

System  (2.  36)  can  be  conceived  as  S systems  of  the  form 

= gjfrj.t)  (2.37) 

additively  interconnected  through  the  S functions  h^(t,  x),  i=  1,...,S. 

Each  unforced  subsystem  (2.  37)  is  assumed  to  be  globally 
asymptotically  stable  and  to  possess  a scalar  Lyapunov  function  from 
which  its  stability  properties  are  evident.  Namely  it  is  assumed  that 
there  exist  functions  v.(x.,t)  such  that 

l 


0;i(||xJ|)  s v.(tfx.)  s 0._(||x.||) 
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1 + + 

/here  0.j(e)  are  increasing,  C functions  from  R to  R and  such 


that  0.^(0)  = 0- j(°°)  = ® • 

In  the  search  for  a Lyapunov  function  for  the  entire  system  (2.  36), 
an  intuitively  appealing  idea  is  to  construct  one  on  the  basis  of  the  v^.  In 
general  this  is  not  possible.  A case  in  which  it  is_  possible  though  can  be 
described  as  follows:  The  crucial  assumption  is  that  the  term  dv./dx  • h^ 
can  be  bounded  by  the  functions  0.3(||x||).  In  particular,  assume  that 


(2.  38) 


and  that  the  are  bounded 


sup  | §..(x,t)|  = 0 < 

x,t 


(2.  39) 


With  the  above  assumptions,  the  following  bound  on  dv^/dt  is  immediate 


dv.  dv.  dv. 

-6T=  -dF+  -5T  (g^tj  + h^t)) 


s - 


0i3(||xill)  + £ §ij<x,  t)  0j3(llxjl 


-0i3(|lxi»)  + J}  9ij  0j3<llxjl 


(2.40) 


We  can  rewrite  (2.40)  in  vector  form 


v = (Vj,  ^ A(0j^(  II  Xj  || tfgjl  ||xs  || ))  (2.41) 

where  A is  the  M-matrix  fa.^j  = {-6..  + 9..]  where  6^  is  the  Kronecker 

6 and  9..  is  defined  in  (2.  39).  Clearly  {a..}  is  an  M-matrix  and  its 
ij  XJ 

stability  is  equivalent  to  the  Hicks  condition^-.  The  stability  of  J is 
related  to  the  stability  of  (2.  36)  by  the  following 

PROPOSITION  2.42.  The  system  (2.  36)  is  globally  asymptotically 
stable  if  A = {a.jj  satisfies  the  Hicks  conditions. 

Proof.  If  A satisfies  the  Hicks  conditions,  there  exists  a positive 
vector  d such  that  d'A  is  a negative  vector  [4].  Consider  the  scalar 


function 


V(t,x)  = d'  • v = £ d.  v (t,x  ) 

i=l 


Then  from  (2.  41)  we  get 


d'v  = V £ dA(013,  023,  . . . ,0S3)  <0 


and  stability  follows  from  Lyapunov  type  arguments. 

The  stability  analysis  of  a n =Si=1  n.  - dimensional  problem 
reduces,  under  the  above  conditions  to  checking  the  Hicks  conditions  for  a 
S X S matrix.  This  simplification  can  be  applied  only  to  systems  satis- 
fying (2.  38),  (2.  39),  which  are  apparently  quite  restrictive.  Still,  there 
are  reports  of  applications  of  this  analysis  to  electric  power  problems 
[9],  which  show  that  the  class  of  systems  examined  above  is  restricted  but 
interesting  for  applications. 

Under  slightly  stronger  assumptions  than  those  of  Proposition  2.42, 
a Metzler  stability  result  can  be  obtained  for  systems  of  the  form  (2.  36). 


i 


Let  us  assume  that  each  subsystem  in  (2.  36)  has  its  particular  rate  of 
adjustment,  i.  e.  , 


xi  = Ki 


i = 1, . . . , S 


(2.43) 


where  K.  = k.(t)  • I,  k.(t)  continuous,  positive  function  of  t and  I a 
ii  l 

n.  X m identity  matrix.  We  assume  as  before  that  there  is  a lower  bound 
0 < kj  ^ k.(t)  for  i = 1,  . . . ,S  . 

Let  us  assume  now  that  the  subsystems  possess  norm-bounded 

Lyapunov  functions  v.(x.)  such  that 

i i 


T).  , lx.  ||  S v.(x.)  S n • X. 
il  i ii  i2  i 


s '"ii 

1 


(2.44) 


and  an  interconnection  bound 


ov.  a 

• h.(x.,t)  * r § (x,t)||x  II 

1 J=1  J J 


(2.45) 


sup  1 5--(x,  t)  | * 0 
x,  t J 


holds . 


Then  the  following  bound  holds  for  dv^/ dt 


dT  = ki(t)  • d3T  (gi  + hi>  55  " kiTli3ll*ill  + £ ki  9ijlxjll 

i j=  l J J 


s - k. 


1 n 


8 0.. 


v.  - £ -11  V. 

i2  1 j=l  V J 


(2.46) 


Writing  (2.  12)  in  matrix  form  we  get 


We  now  have 


PROPOSITION  2.49.  Under  assumptions  (2.44),  (2.45)  all  the 


systems  described  by  (2.43)  are  stable  provided  A is  a stable  matrix 


Proof.  A is  a stable  M-matrix.  Hence  there  exists  a positive 


diagonal  matrix  D=  diagfd j,  . . . , dg  J such  that  DAD  satisfies,  according 
to  the  result  already  quoted  in  Proposition  2.13  from  [8],  the  inequalities 


Let  now 
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Along  a trajectory  of  any  system  in  (2.43)  the  v^(x(t))  are  continuously 
differentiable  functions  of  t.  Using  the  same  arguments  that  were  used  in 
the  proof  of  Proposition  2.  13  we  can  show  that  the  right  hand  derivative 
of  v(x(t))  exists  everywhere  and 


max 

j€J(t) 

d.  - v.(x(t)) 

max 

j€J(t) 

k.  a..(d.v.)  + 
3 JJ  3 3 

E 

i/j 

d. 

a.,  -r1 
J1  d. 

s - k • TJ  • v(x(t)) 


where  k = min{kj(t)J.  As  in  Proposition  2.  13  we  can  conclude  that 


v(x(t))  ^ exp( -k77t)v(x(0))  and  hence  v(t)  -»0,  x(t)  -*  0. 


Q.  E.  D. 
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III.  THE  APPLICATION  OF  HJB-INEQUALITIES  TO  OPTIMIZATION 


1.  Introduction 

In  this  Section  we  examine  some  issues  in  large  scale  optimization. 
We  consider  a collection  of  S control  systems 


x = f.(x.,  u.,t)  ; 


n. 


Xi(0)  = Xi0 


(3.  1) 


where  x.  € R 1 for  i = 1,  . . . ,S  and  the  cost  functional 


= E -V*!’ Ui} 

i=  1 


S 

E 

i=i 


T 

/ 


Li(x.,ui;t)  dt  + 0i(x.T) 


(3.2) 


I 


i 

I 


i 

k ' 


It  can  be  seen  that  finding  the  infimum  of  (3.  2)  over  the  trajectories  of 
(3.  2)  decouples  to  S lower  dimensional  problems,  namely 
T 


inf  / L.(x.,u.,t)  dt  + 0.(x._) 

•/,  111  1 1 J. 


Ui  fc0 


(3.3) 


x.  = f.(x.,u.,t)  x.(0)  = x 

i i i’  i’  i 


iO 


The  decomposition  of  problem  (3.  2)  will  be  invalid  the  moment  any 

coupling  is  introduced  among  the  subsystems  (3.  1).  Let  x = (x.,.  . . ,xc) 

n. 

€ Rn  be  the  total  state  vector  and  g.(x,t)  : Rn  x [tg,  T]  -»  R 1 be  a 
continuous  function.  Then  the  coupling  between  the  subsystems  can  be 
modelled  additively  as 


x.  = f.(x.,  u.;t)  + g.(x,t) 


(3.4) 
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Naturally,  (3.4)  is  not  the  most  general  form  of  coupling  between  sub- 
systems. 

We  are  interested  in  comparing  the  optimization  problems 
corresponding  to  the  decoupled  system  (3.  1)  and  the  coupled  one  (3.4). 
More  generally,  we  will  consider  the  optimization  of  the  functional 


J(x0’u5t0)  = J L(x,  u;t)  dt  + <£(xt) 


over  the  trajectories  of  the  systems 


x = f(x,u;t) 


x(t0)  = xQ 


x = f(x,u;t)  + g(x;t)  x(tQ)  = xQ  (3.7) 

where  x € Rn,  u € Rm.  Two  questions  become  of  interest  in  this  context. 
First,  is  it  possible  to  compare  the  payoffs  corresponding  to  systems  (3.  6) 
and  (3.  7)  without  explicitly  solving  for  the  optimum  of  the,  presumably, 
complicated  system  (3.  7)?  Second,  how  good  are  "naive"  optimization 
policies  for  (3.  7)?  If,  say,  the  policy  u(x,  t)  that  has  been  proven  optimal 
for  (3.6)  is  applied  to  (3.  7)  how  good  is  it  going  to  be?  We  provide 
conditions  under  which  it  is  possible  to  answer  such  questions. 

The  techniques  used  in  this  Section  is  that  relating  to  inequalities  of 
the  Hamilton  Jacobi  Bellman  type  (henceforth  abbreviated  as  HJB) 
inequalities.  J.  C.  Willems  [8,  9]  has  used  similar  inequalities,  which  he 
calls  dissipation  inequalities,  in  stability  theory.  We  hope  to  show  here 
that  they  are  as  interesting  in  optimization  theory.  These  inequalities  play 
an  important  role  in  a class  of  iterative  optimization  techniques  introduced 


by  Bellman,  a special  case  of  which  is  Kleinman's  algorithm  [5]  for  the 
solution  of  the  algebraic  Riccati  equation.  In  the  last  section  of  this 
chapter  we  extend  the  applicability  of  these  algorithms  through  the  use  of 
HJB  inequalities.  The  extended  form  of  the  algorithm  will  be  needed  in 
Section  4. 


Hamilton  Jacobi  Bellman  (HJB)  Inequalities 


Consider  first  a finite  time  optimization  problem,  namely 


1 

Inf  I L(x,  uj 
u 


t)  dt  + <^(xT) 


subject  to  dynamics  and  initial  condition 


x = f(x  ,uj  t) 


x(t0)  = x0 


and  the  control  constraint  u(-)  £ 2/ 


= ju:Rnx  [t0,T]-flc  Rm  and 


piecewise  continuous 


(3.  1C 


Let  V(x,  t)  be  the  infimum  value  of  the  objective  function  over  the  interval 
[t,  T ] starting  at  state  x at  time  t.  The  following  theorem  states  that 
V(x,  t)  can  be  characterized  through  a partial  differential  equation,  the 
Hamilton- Jacobi  - Bellman  equation. 


THEOREM  3.  11  [7,  p.  192]  Suppose  there  exists  a differentiable 

function  V : Rn  X [t^jT]-*  R satisfying 
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+ Min  | L(x,  u,  t)  + |—  • f(x,  a;  t)  | u € 0 j = 0 


(3.  12) 


and  the  boundary  condition 


V(x,  T)  = 0<x) 


Suppose  there  exists  a function  u : R x [tQ,T]  -•  piecewise  continuous 
in  t and  Lipschitz  in  x satisfying 

L(x,u;t)+  g f(x,u;t)  = Min  | L(x,  u;  t)  + 0 f(x,  u;  t)  | u € Ci  J 

Then  u is  an  optimal  feedback  control  for  the  problem  (3.8)  and  V is  the 
optimal  value  function. 

It  is  of  interest  to  consider  the  inequalities  corresponding  to  (3.  12). 
We  do  so  in  the  following 


PROPOSITION  3.  13.  (i)  Let  us  assume  that  the  statement  of 

Theorem  3.  11  is  satisfied.  Suppose  there  exists  a differentiable  function 
U : Rn  X [tg;  T]  -»  R satisfying 


+ Min  | L(x,  u;  t)  + ~ • f(x,  u;  t)  | u £ 0 j is  0 


(3.  14) 


and  the  boundary  condition 


U(x;T)  ^ 0(x) 


(3.  15) 


t 

i i I 


Then  U(x,  t)  £ V(x;  t)  for  (t,  x)  6 [t^,  T]  x R . 

(ii)  As  in  (i)  let  there  be  a differentiable  U(x,t)  and  a u(x,  t)  € <W 


such  that 


jiShti  iiii’i'i  t 


+ L(x,  u;t)  + |^-  f(x,  u;t)  s 0 


(3.  16) 


U(x;T)  s 0 (x) 


Then  U(x,  t)  £ V(x,  t)  for  (x,  t)  € Rn  X [t^,T], 


Proof.  Consider  the  value  of  the  cost  functional 


tjUjtj)  = J* 


L(x,  u,t)  dt  + 0(xT> 


(3.  17) 


when  the  control  law  u(x,t)€  U is  applied  to  the  system 


x = f(x,  u(x,  t),  t) 


with  initial  condition  x(tj)  = x^  . 


We  now  add  and  subtract 


U(x,  t)  dt 


(U  is  computed  along  the  trajectory)  to  J(xj,u,  tj): 

T T 

J(XpU,  tj)  = f L(x,  u,  t)  dt  + 0(xT)  + f I ~ ^ U(x,  t)+  ^ f(x,  u,t)jdt 

t -i  t i J 


= U(x, , t. ) + 0(Xrr)  - U{x_,,  T)  + f I|?  + L+  I?  £ 1 


3.  18 


is  nonnegative.  The  difference  0(xT)  - U(xT>  T)  is  also  nonnegative  by 
(3.  1 5 ) ? and  thus  (3.  18)  implies  that 


J(Xj,u,tj)  £ Ufxptj) 


for  any  control  law  u(x,  t)  and  any  Xp  tp  Since  V(xj,tj)  = min^  J(xp  u>  tj) 
it  follows  that  V(xj,tj)  ^ Ufxptj),  which  proves  part  (i). 

To  prove  part  (ii),  assume  that  (3.  16)  and  (3.  17)  hold.  This  time  we 
can’t  conclude  from  (3.  16)  that 


9U  + L(x,  u,  t)  + |^  f(x,  u,  t)  SO 


at 


for  arbitrary  x,  u;  Equation  (3.  16)  states  tiat 


+ L(x,u,t)  + |^  f(x,  u,t)  SO 


■H 


.1 
i 1 


5 
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To  prove  part  (ii)  we  consider  the  value  function  J(xp  u(x,  t),  tj)  with  u 
defined  above.  We  will  show  next  that  J(xj,u,  tj)  s U(xptj)  and  hence 
Vlxptj)  s J(x,u,tj)  s Utxptj). 

Looking  at  (3.  18)  with  u(x, t)  = u(x,t)  we  get  that  the  integral 
T 


/ 


9U  + L(x,  u,t)  + f(x,u,  t) 


at 


dt 


is  nonpositive  and  so  is  0(x_)  - U(x_,  T).  It  follows  from  (3.  18)  that 


J(xp  u,  tj)  s U(xptj). 


Q.  E.  D. 


Remark.  In  the  above  Proposition  we  can  replace  the  min  operator 
by  any  of  max,  inf,  sup  operators  without  any  significant  change  in  the 
results. 

For  the  class  of  infinite  duration  optimization  problems,  similar 
results  have  been  derived  by  Willems  [9].  In  particular,  consider  the 
following  optimization  problems 

m 

V,(x.)  = inf  J L(x,  a)  dt 
« *0 
CD 

V(x~)  = inf  / L(x,  u)  dt;  lim  x = 0 

0 u t-® 


-I 


\.  I 


V (x_)  = inf  / L(x,  u)  dt 

” ° “-T  \ 


all  subject  to  the  dynamic  constraints 


x = f(x,u) 


x(t0)  = XQ  ; U 6 u 
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t I 


Clearly  Vn(xQ)  £ V£(xQ)  £ V (xQ ) for  all  xQ.  We  have  the  following 
proposition  relating  V,  V£,  to  the  solutions  of  a time  invariant  HJB 
inequality. 


PROPOSITION  3.  20.  Consider  the  HJB  inequalities 


inf 


au 


L(x,  u)  + f(x,  U)|u6  n 


;>  o 


(3.21) 


inf 


— 3 v — . 

L(x>u)  + -^  f<x,u)|u€fl 


£ 0 


(3.  22) 


and  suppose  that  there  exists  a nonpositive  function  V£(x)  satisfying  (3.21) 
a nonnegative  function  V2(x)  and  a u(x,  t)  satisfying  (3.  22).  Then 


Vj(x)  £ Vn(x)  £ Vf(x)  £ V2(x)  (3.23) 

Furthermore,  if  V3(x)  satisfies  (3.21)  and  V3(0)  = 0,  we  have 

V3(x)  £ V(x) 


Proof.  Consider  the  integral  £ L(x,  u)  dt  subject  to  (3.  19)- 


Adding  and  subtracting  to  it 

T, 


r 


V.  (x)  dt 


(i  = 1,2,3) 


we  get 


SI 


1 


4 

I i 
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For  i = 1,  equations  (3.  21)  and  (3.  24)  imply  that  J(xQ,  T)  for 

any  x^,  any  T and  any  control  u,  hence  Vj (x^)  s V^(xq).  Similarly  for 
i = 2 we  get  V^Xq)  S ^(x^)  considering  a control  law 
that  satisfies  (3.  22).  Finally  for  i = 3 and  T —*>  we  get  from  (3.  24)  that 


inf 

u 


lim 
T-» 
x(T)-« « 


J(x0; T) 


V<x0)*V3(x0) 


Q.  E.  D. 


Remark.  This  proposition  is  due  to  Willems  [9]  except  for  the  parts 
relating  to  (3.  22). 


3.  Comparison  of  Systems 


3.  1 General  Results 

We  now  consider  our  original  problem,  to  compare  the  infima  of  the 
functional 


fc(j) 


L(x,  u;t)  dt  + 0(xT) 


(3.  1) 


over  the  trajectories  of  the  systems 


x = f(x,  u;t)  x(tQ)  = Xq  (3.2) 

and 

x = f(x,  u;t)  + g(x,  t)  x(tQ)  = xQ  (3.3) 


Let  us  assume  that  the  value  functions  corresponding  to  these  problems  are 
Vj(x;t),  V^(x;t).  We  have  the  following  comparison-type  proposition 
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PROPOSITION  3.  25:  Suppose  that  Vjtxjt),  V2(x;  t)  are  continuously 

differentiable,  satisfy  the  corresponding  Hamilton- Jacobi -Bellman 
equations 

■|^+  min  j L(x,  u;t)  + -g— *■  * f(x,  u;  t)  | u 6 n | = 0 (3.26) 

and 

•|^+ min  I L(x,  u;t)  + g^*  [f(x,  u;  t)  + g(x;  t)]|  u 6 0 I = 0 

' ’ (3.27) 


and  the  common  boundary  condition  V(x;  T)  = 0(x).  If 


9V1 

-fo"  • g(x;t)  s 0 


for  all  x(Rn,  t € [tQ,  T],  where  g(x;  t)  is  the  "modification  term"  in 
(3.  3)  we  have  Vj(x;  t)  :*  V2(x;  t).  If 

avi 

• g(x,t)  s 0 


then  Vj(x;t)s  V2(x;t)  . 

Proof.  Substituting  Vj(x;t)  in  (3.  27)  we  get 


avi  ( 9V1  ) 9V1 

-g^+min  (L(x,  u;t)+  f(x,  u;  t)  | u € | + -gj-  * g(x,  t) 


g(x,t)  s 0 


Hence  Vj(x,  t)  satisfies  a HJB  inequality  and  by  Proposition  3.  13, 

V.(x;t)  :>  V (x;t).  The  second  statement  of  the  proposition  follows  in  the 

Q.E.  D. 


same  manner. 


Similar  comparison  results  can  be  derived  from  Proposition  3.  20. 


inf  I L(x,  u)  dt; 

lim  x(t)  = 0 

u 4o 

t-»  00 

f(x,  u)  x(tg)  = 

xo 

inf  J L(x,  u)  dt; 

lim  x(t)  = 

u tQ 

t-»  00 

f(x,  u)  + g(x) 

1 2 

Assume  furthermore  that  V , V are  continuously  differentiable  and 
satisfy  the  HJB  equations 


( av1  ) 

iin  |L(x,  u)  + f(x,  u)  j u € ft  j = 0 


min  L(x,  u)  + [f(x,  u)  + g(x)]|  u g 0 > = 0 


1 2 

as  well  as  the  boundary  condition  V (0)  = V (0)  = 0.  We  then  have 


PROPOSITION  3.30:  The  inequality 


8 V 

-K-  s(x)  * 0 
1 2 

implies  that  V (x)  £ V (x). 


1 2 
Proof.  Substitute  V (x)  in  the  HJB  equation  satisfied  by  V (x) 


and  note 
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inf  | L(x,  u)  + [f(x,u)  + g(x)]|u  € 0 


= inf  <L(x,u)  + -4-  f(x,u)|u  €0  J+  g(*> 


g(x)  £ 0 


1 2 

It  follows  from  Proposition  3.  20  that  V (x)  £ V (x). 


3.  2 Linear  Quadratic  Problems 


It  is  well  known  [4]  that  for  the  linear  quadratic  minimization 


Q.E.  D. 


problem 


T 

nin  J x1 

u 


C'Cx  + u'u  dt 


(3.31) 


x = Ax  + Bu  x(tQ)  = xQ 

with  (A,  B,  C)  a minimal  realization,  the  optimal  value  function  is 
V^XqiV  = xoKi<to;T’0)xO  where  Ki(t;T>°)  is  the  solution  of  the 
Riccati  equation 

Kj  + A'Kj  + KjA  + C'C  = KjBB'Kj  ; K^TjT.O)  = 0 (3.32) 


Furthermore  lim^  K.(t;T,0)  = k|  , the  unique  stabilizing  solution 

“i  1 


of  the  Algebraic  Riccati  equation  (ARE) 


A'Kj+KjA  + C'C  = K’BB'Kj 


(3.  33) 


Similar  results  hold  when  the  system  matrix  A is  replaced  by  A + H.  We 


K (tn;  T,  0)xn  and  the  steady 


denote  the  value  function  as  V_(x 


state  value  matrix  by  lim—^^K^t;  T,  0)  = K^. 

Using  Proposition  3.  25  we  get  that  the  condition  x’K^(t;  T,  0)Hx  &0 
(£  0)  or  equivalently 


implies  that  K^ftj  T,  0)  :>  K^(t;  T,  0)  (s  ). 

Proposition  3.  20  implies  a similar  result  for  the  steady  state  values 


3.  3 Bounds  for  Suboptimal  Controllers 


We  now  consider  the  problem  of  characterizing  the  performance 
of  suboptimal  controllers  for  interconnected  systems  that  was  posed 
in  the  introduction.  Bailey  and  Laub  [1]  consider  the  following  situation 
Assume  that  for  the  optimization  problem 


there  exists  an  optimal  value  function  Vj(x;t)  and  a piecewise  continuous 
control  u,(x;t)  satisfying  the  relation 


the  boundary  condition  V(0;t)  = 0 and 


wwscss*- 
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(x,  t)  = arg  max 


9V1  1 

L(x,  a;  t)  + f(x,  u,  t)  | u€  0 


Assume  furthermore  that  f(0,u;t)  =0;  so  x = 0 is  an  equilibrium  point 
of  (3.  34).  Bailey  and  Laub  consider  what  will  happen  if  the  control 
Uj(x,  t)  is  applied  to  the  modified  system 


x = f(x,  u;t)  '+  g(x,  u;t) 


x(to)  = xQ 


(3.  36) 


(with  the  assumption  g(0,u;t)  = 0,  so  that  x = 0 is  an  equilibrium  of 
(3.  36)).  They  show  that  if  the  condition 


8V, 


L(x,  UjJt)  - -gj-  • g(x,Ui;t)  i 0 


(3.  37) 


holds  for  all  x,  t then 

i)  The  suboptimal  control  Uj(x,  t)  when  applied  to  (3.  36)  results  in 
a trajectory  that  converges  to  0 as  t 

ii)  If  Vg(x^;tQ)  is  the  payoff  that  results  when  the  control  law  u^ 
is  applied  to  (3.  36)  then 


Vs(x;t)  s q V X(x;  t) 


where 


q = sup 
T^t 


L(x,  Up  t) 

L(x,UpT)-  g(x,UpT) 


< 00 


The  above  upper  bound  is  of  interest  in  characterizing  the  performance  of 
the  suboptimal  controller.  Bailey  and  Laub  point  out  the  need  for  a lower 


-6 


bound  on  its  performance.  Conditions  for  such  a bound  can  be  easily 
derived  using  arguments  of  this  section. 

Let  us  write  Vg(x;t)  as 


00 

vs^xo;to^  = f L(x>ui;t)  dt  + 


00 

f -Ay 

J dt  1 


aVj  avj 


+ ~ar+  ~W  tf  + dt 


= vi(xo;to}  " Vl(xT’T)  + J Ivit  + L(x 

T-»®  ( tQ 


J[vlt  + L(x,ui;t)+^  f] 


av 


dx 


C avi 

V1<VV+  J ~dx  g^x>ul>to^  dt  • 


The  last  equality  follows  from  the  fact  that  limm  =0.  V,  is 

T-*«  T 7 1 

continuous  and  that  Vj  satisfies  a HJB  equation.  Therefore,  if  we  require 
in  addition  to  the  Bailey- Laub  condition 

9V1 

L(x,Upt)  s -gj  g^Ujjt)  (3.37) 

that 

avi 

L(x,upt)  & g(x,  Ujjt)  so  (3.38) 

we  obtain  an  upper  as  well  as  a lower  bound  on  the  performance  of  the 
suboptimal  controller  Up  namely 


q V j(x,  t)  S Vg(x,  t)  S V j(x,  t) 


(3.  39) 
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4.  Optimization  Algorithms 

4.  1 Bellman's  Algorithm 

In  1954  Bellman  [3]  proposed  a monotone  policy-updating  algorithm 

for  optimization  problems.  "Monotone"  means  that  every  iteration  of  the 

algorithm  results  in  a global  improvement  of  the  value  function.  A brief 

exposition  of  the  algorithm  would  be  as  follows.  Consider  again  the 

optimization  problem  at  the  beginning  of  this  section.  Find 

T 

inf  / L(x,  u;t)  dt  + 0(xT) 
u *tQ 

on  the  trajectories  of  the  system 


x = f(x,  u;t)  x(tQ)  = xQ 


Assume  that  the  piecewise  continuous  control  law  u^(x,  t)  gives  rise  to  the 
differentiable  value  function  V.(x,t).  If  x(t)  is  the  resulting  trajectory 
then  Vj(x(t),  t)  is  a function  of  t and  we  have 


dV. 


dV. 


dt  Vi  = -L(x>Vt)  = ~&r  + ~dT  f(x>  ui> 


(3.40) 


Consider  now  the  control  law 


u i(x,t)  = arg  min 


9V. 


L(x,u,t)+  • f(x,  u;t)ju€^ 


(3.41) 


Assume  that  ui+j(x,t)  is  piecewise  continuous  and  defines  a value  function 
V.+  j(x;t),  which  also  satisfies  an  equation  similar  to  (3.40). 


r 
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av. 


i+i 


sr 


dyi+i 


+ L(x,  u.  + 1;t)  + -3—  • f(x,u.+1;t)  = 0 


(3.42) 


Furthermore  by  (3.  40)  and  the  definition  of 


8V.  dV. 

r + L(x,u.  + 1;t)  + f(x,u.  + 1;t)  * 0 


at 


dx 


(3.43) 


Subtracting  now  (3.42)  from  (3.43)  we  get 


8(Vi"Vi+l)  a(Vi*Vi+l} 

- V-  1 + - - 'axJ — f(x>ui+i’t}  s 0 


and  hence 


a<Vvi+i> 


at 


+ min 


u 


a^vi-vi+i) 

9x 


f(x,  u J t)  £ 0 


(3.44) 


Now  (3.44)  is  a HJB  inequality  corresponding  to  an  optimization  problem 
with  payoff  L = 0.  Hence  by  Proposition  3.  13  V.  2 V^+1  which  shows 
that  the  control  law  is  better  than  u^  provided  that  it  is  admissible, 

i.  e.  , piecewise  continuous.  If  all  assumptions  hold,  the  above  computational 
procedure  yields  a sequence  of  value  functions  Vq  ^ Vj  ^ s . . . which  are 
monotone  decreasing. 

The  above  policy -iteration  algorithm  would  provide  an  effective  way 
to  solve  any  optimization  problem  provided  that  the  sequence  V^x,  t) 
converges  to  the  optimal  value  function  V(x,t).  This  is  not  true  in  general. 
However,  it  is  true  for  the  linear  quadratic  problem,  which  we  examine  in 
detail  in  the  next  section.  First  we  would  like  to  examine  the  application  of 
Bellman's  algorithm  to  the  stability  constrained  infinite  duration  problems. 


Find 


Let  us  assume  that  the  stabilizing  control  law  u.  (x,  t)  results  in  the 
value  function  V.(x).  As  in  the  proof  of  (3.40),  it  can  be  easily  shown  that 


Let  us  assume  furthermore  that  the  control  law  obtained  from  Bellman's 


algorithm  is  admissible,  i.  e.  piecewise  continuous  and  stabilizing 


u ,(x,t)  = arg  min  L(x,  u)  + 


and  it  gives  rise  to  a value  function  V 


We  make  the  further  assumption  that  V.(0)  = V 


in  (3.48)  is  nonpositive  by  the  definition  of  u 


The  above  shows  that  Bellman's  algorithm  will  be  monotonically  convergent 
even  in  the  infinite  duration,  stability  constrained  case  provided  that  we  can 
guarantee  that  the  control  laws  generated  at  each  step  of  the  algorithm  are 
admissible,  i.  e.  stabilizing.  This  happens  to  be  the  case  in  the  most 
important  such  algorithm,  Bellman's  algorithm  for  the  linear  quadratic 
minimization  problem  which  we  examine  next. 

4.  2 Algorithms  for  the  Linear  Regulator  Problem 

The  application  of  the  policy  iteration  algorithms  to  the  regulator 
problem.  Find 


inf  J x'Qx  + u'u 

dt; 

lim  xT  - 0 

(3.49) 

0 

T— *00 

x = Ax  + Bu 

x(0) 

= x„  ; (A,  B)  controllable 

is  particularly  enlightening.  It  hinges  on  the  fact  that  the  optimal  value 
function  is  quadratic  in  the  state,  and  hence  the  algorithm  proceeds  by 
updating  the  parameters  of  a quadratic  form  in  the  way  stipulated  by 
Bellman's  algorithm.  Starting  from  a stabilizing  control  law  u^  = L^x 
i.  e.  such  that  Re  X.(A  + BL.Q)  < 0,  the  resulting  value  is  VQ(x)  = x^KQx0 
where 

(A  + BL0)'K0  + Kq(A  + BLq)  = - Q - 
The  policy  minimization  step  suggests  to  use 

I 

Uj  = arg  min  {xQx  + u'u  + 2x'Kq(Ax  + Bu)J 


In  general,  the  updating  procedure  is 


u.,,  = - B'K.x  (3. 

l+l  1 

(A  - BB'K.)'K  , + K . (A  - BB'K.)  = -Q  - K.BB'K.  (3. 
l l+l  l+l  i ii 

The  algorithm  in  equations  (3.  50)  and  (3.  51)  was  derived  by  Puri  and 
Gruver  [6]  from  policy  iteration  considerations  and  by  Kleinman  [5]  who 
derived  it  by  applying  Newton's  root  finding  procedure  to  the  Riccati 
equation 

A'K  + KA  + Q = KBB'K  (3. 

In  both  approaches,  the  crucial  step  of  the  algorithm  is  to  show  that  the 
updating  of  the  control  laws  is  valid,  i.  e.  it  gives  a stable  control  law. 
The  arguments  presented  to  that  effect  in  both  references  are  vague.  V 
thus  propose  to  give  independent  proofs , and  in  the  process  provide  a 
characterization  of  the  class  of  problems  for  which  this  algorithm  can  be 
successfully  applied. 

We  first  consider  the  case  where  the  cost  matrix  Q is  positive 
definite. 

PROPOSITION  3.53:  Consider  the  regulator  problem  (3.49)  with 

Q = Q'  > 0.  Then 

i)  There  exists  a unique  solution  K+ > 0 to 

f 

A'K  + KA  + Q = KBB’K  (3. 

such  that  Re  X(A  - BB'K+)  < 0.  The  control  law  u = -B'K+x  is  the 
solution  (3.49)  and  V(x)  = x'K+x  is  the  optimal  value  function. 


ii)  The  sequence  KQ,  Kj,  . . . , Kr, . . . defined  by  equations  ( 
(3.  51)  is  well  defined,  monotonically  decreasing  and  converges  to  K 
Each  u.  = - B'K.x  is  a stabilizing  control  law. 


Proof.  Part  (i)  is  well  known  [4]  so  for  part  (ii)  we  proceed  by 

Lx  be  a stabilizing  control  law  and  let 


induction, 


V(x)  = x'Kx,  K = K'  be  the  corresponding  value  function.  The  assumption 
Q>  0 guarantees  that  K>0.  Also,  K satisfies  the  Lyapunov  equation 


Consider  now  the  updated  control  law 


We  now  try  V = x'Kx  as  a Lyapunov  function  for  (3.  56).  We  have  along 
the  trajectories 


The  derivative  is  nonpositive,  and  in  fact  it  is  negative  for  x/0.  Thus 
the  Lyapunov  function  V(x)  = x'Kx  implies  the  stability  of  (3.  56). 

The  iteration  implied  in  (3.  50),  (3.  51)  is  admissible  and  gives  rise 


to  a monotone  decreasing  sequence  of  matrices  K.,  and  K.  s K 


lim  K.  = K exists  and  satisfies  (3.  54),  as  can  be  seen  by  taking 
i->  » 1 

limits  in  (3.51).  Thus  K iK  >0,  which  contradicts  the  uniqueness  of 
the  positive  definite  solution  of  (3.  54)  unless  K = K+.  Hence  lim.^^Kj  = K 

Q.  E.  D. 

The  assumption  Q>  0 is  fairly  restrictive.  Linear  quadratic 
problems  in  an  indefinite  Q will  arise  in  the  problems  of  stabilization 
and  game  theory  to  be  considered  in  Sections  IV  and  V.  The  algebraic 
Riccati  equation 

A'K+KA  + Q = KBB’K  (3.57) 

will  play  a crucial  role  in  their  solution.  It  becomes  therefore  important 
to  establish  the  validity  of  the  Bellman  algorithm  for  the  solution  of  the 
(ARE)  (3.  57)  with  a general  Q matrix.  Computational  experiments, 
performed  on  a Wang-2200  minicomputer,  have  shown  that  the  algorithm 
almost  always  converges  monotonically  to  a solution  of  the  (ARE)  (3.  57) 
(provided  of  course  that  a solution  to  (3.  57)  exists.  ) We  can  give  a 
theoretical  justification  of  these  experimental  results.  The  numerical 
experiments  are  given  in  Appendix  2.  To  prove  the  monotonic  convergence 
of  the  algorithm  we  make  strong  use  of  the  theory  of  the  solutions  of  the 
algebraic  Riccati  equation.  A summary  of  that  theory  is  provided  in 
Appendix  1.  In  fact  our  main  result  requires  a mild  condition  on  the 
maximal  and  the  minimal  solutions,  K+  and  K of  the  Riccati  equation 
(Theorem  A.  1.  1 of  Appendix  1).  This  result  is  stated  in 


is  controllable,  and  that 


there  exists  a real  symmetric  solution  of  the  (ARE) 


Assume  furthermore  that  K >K  . Then  Bellman's  algorithm  as  expressed 


in  equations  (3.50),  (3.51)  gives  rise  to  a decreasing  sequence 
K j > > . . . > K.  > . . . which  converges  to  K+  provided  of  course  that 

the  algorithm  is  started  with  a stabilizing  control  un  = Lx,  i.  e. 


To  prove  Proposition  3.  5 8 we  need  the  following  lemmas 


LEMMA  3.  59 


Let  K be  a real  symmetric  solution  to  the  (ARE) 


Then 


Proof.  This  is  a special  case  of  the  well  known  lemma  of 
"completing  the  square",  for  instance  Lemma  6 of  [9]. 

The  following  lemma  states  that  Bellman's  procedure  for  improving 
a given  control  law  is  effective  even  when  the  improved  control  law  is 
applied  for  a subinterval  of  the  entire  optimization  horizon.  In  particular, 
consider  the  infinite  horizon  cost  functional  of  problem  (3.49) 


00 

(“iV  “ / 


x'Qx  + u'u  dt 


x = Ax  + Bu  x(0)  = x. 


and  assume  that  the  stabilizing  control  law  u^  •=  Lx  is  applied.  Then 


J(Lx;x0)  = x^Kqx;  Kn  = K'n 


0 0 


where  KQ  satisfies 


(A  + BL)'K0  + Kq(A  + BL)  = - Q - L'L 


Consider  now  the  improved  control  law 


Uj(x)  = arg  min  fx'Qx  + u'u+  2x'Kq(Ax  + Bu)}  = 


" B'Kox 


and  let  us  apply  the  control  Uj(x)  on  the  time  interval  [0,  T]  and  the 
control  Uq(x)  on  (T,»),  where  T is  a fixed  but  arbitrary  time.  Call 


this  piecewise  linear  law  u^(x)  and  J(u,j,;Xq)  the  value  that  results  when 
it  is  applied.  We  claim  that  uT  is  an  improvement  on  uQ  regardless 
of  T. 


LEMMA  3.  61:  J(xQ,uT)  <:  J(X(),  uQ)  = x^KQxo  . 

Proof.  To  simplify  the  notation  let  V(x)  = J(x;Uq).  Along  the 
trajectories  of 

x = Ax  + BUq 

V(xt)  becomes  a time  function  and 

^ V(x)  = - xQx  - u^uQ 

and  hence 

dV 

x'Qx  + UqUq  + (Ax  + BUq)  = 0 (3.  62) 


= (Ax  + Bu0) 


Since  u^(x)  is  defined  as  the  minimum  of  the  above  expression,  it  follows 
that 


9V 


x'Qx  + u^Uj  + (Ax  + Bu^)  £ 0 


(3.63) 


Consider  now  Jtu^jx^).  We  have 


J(xQ;uT)  = 


^ x'Qx  + u'jUj  dt  + J x'Qx  + UqUq 


dt 


/ f-4 


dV 


dt  V(X)  + to  (AX  + BU1} 


dt 


The  second  integral  equals  V(x„)  and  the  term 
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% 

J - ^ V(x)  dt  = V (xQ)  - V(xT). 


Hence 


J(xQ;uT) 


= V (xQ)  + £ jx'Qx  + u'jUj  + |^  (Ax  + Bu^j  dt 


From  (3.  63)we  conclude  that  the  integrand  is  nonpositive  and  hence 


j(xQ;uT)  s v(xQ)  = J(x0;uQ) 


Q.  E.  D. 


Proof  of  Proposition  3,  58.  It  suffices  to  show  that 

u , = arg  min  {x'Qx  + u'u  + 2x’K.(Ax  + Bu)} 

i+1  u 

gives  rise  to  a stable  control  law.  It  would  then  follow  that  the  sequence 

K defined  in  (3.  51)  is  nonincreasing  and  bounded  below  by  K . Hence 
i 

lim  K.  = K & K+  and  K is  a solution  of  (3.  57),  as  can  be  seen  by 

i-*«o  j 

taking  limits  in  (3.  51).  This  contradicts  the  maximality  of  K 
and  hence  Urn.  K = K = K . 

1— • CD  1 

To  show  that  the  control  improving  procedure  results  in  stable 
control  laws  we  consider  as  in  the  remarks  preceding  Lemma  3.61,  the 
stabilizing  control  uQ  = Lx,  the  updated  control  Uj  = -B'Kx  and  the 
piecewise  linear  law  uT(x).  We  showed  in  Lemma  3.61  that 
J(xQ;  UT)  s XqKxq.  We  now  bound  J(uT;  xQ)  from  below  as  follows . Let 

T 


j(xo;ut}  = I*  xQx  + uiui  dt  + l *,Qx  + »o“< 


dt 


(3.64) 
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From  Theorem  A.  1.  4 


x'Qx  + u^uQ  dt  s xtK  xt 


From  Lemma  3.  59 


x'Qx  + u'jiij  dt 


*oK  xo  ■ xtK  xt  + 


/ IK 


+ B'K"x|r  dt 


In  view  of  (3.  65)  and  (3.  66)  we  obtain  from  (3.  64) 


r 

J(x0;uT)  ;>  xQKx0  + x^K*  - K")xt  + / lluj+B'K' 


x dt 


and  finally  using  the  bound  J(xQ;  uT)  s XqKxq 


xq(k~k")x0  ^ x^(K  - K‘)xt  + 


b'1 


+ B'K'xIr  dt 


xq(K  - K“)x0  :>  x^,(K+  - K")xT 


Inequality  (3.  67)  holds  for  any  T.  The  assumption  Kr  - K >0  implies 
that  || x^, ||  is  bounded  for  all  T and  thus  the  control  law  Uj(x)  = - B'Kx 
can't  give  rise  to  unbounded  trajectories. 

To  show  that  u^  is  a stable  law  it  remains  to  show  that  A - BB'K 
can't  have  imaginary  eigenvalues.  If  it  did,  the  linear  system 
x = (A  - BB'K)x  would  have  a periodic  solution  for  some  initial  condition 
x(0)  = Xq.  Let  the  period  of  that  solution  be  T^.  Consider  now  our 

aft 

The  case  of  a zero  eigenvalue  can  be  treated  similarly. 


minimization  problem  with  initial  condition  and  let  the  piecewise 

linear  control  u^,(x)  be  applied  with  T an  integral  multiple  of  the  period 
Tp  say  T = nTj.  In  the  proof  of  Lemma  3.  61  we  proved  the  expression 


J(x, 


uT) 


= V<x0) 


T 

II- 


x'Qx  + 


ulul 


9V 

9x 


(Ax  + 


Buj) 


dt 


where  the  integrand  is  a nonpositive  quantity. 


From  the  periodicity  assumption  this  expression  can  be  written  as 

T, 


J(x0’UT)  = Y(: 


r°,  + n { t 


9V 


x'Qx  + UjUj  + (Ax  + Buj) 


dt 


We  claim  that  the  integral  has  to  be  zero.  Otherwise  it  would  be  a negative 
number  and  since  n is  arbitrary,  J(XqJu^)  could  be  made  smaller  than 
x qK  Xq  for  sufficiently  large  n,  contradicting  the  fact  that 
inf^  J(x;  u)  = x'K+x  (Theorem  A.  1.  4).  The  integral  is  therefore  zero  and  so 
is  the  integrand 

dV 

x'Qx  + u^Uj  + (Ax  + Bu^)  = 0 

(Here  we  use  the  fact  that  the  integrand  is  nonpositive).  We  know  that  u^ 
is  the  unique  control  that  minimizes  the  expression 

9V 

x'Qx  + u'u  + (Ax  + Bu) 

Also,  the  above  expression  becomes  zero  for  u = Uq  as  was  shown  in  the 
proof  of  Lemma  3.  61.  The  minimum  is  thus  attained  by  u^  and  hence, 
by  the  uniqueness  of  the  minimum,  Uj(x)  = Uq(x)  for  all  x on  the  periodic 
trajectory  of  x = Ax  + Buj(x).  The  system  x = Ax  + BUq(x)  gives  rise 
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to  the  same  periodic  trajectory  as  x = Ax  + Bu^(x)  if  started  at  the  initial 
condition  x(0)  - xq-  This  is  a contradiction  to  the  assumption  that  is 
a stabilizing  control.  Q.  E.  D. 

Remark.  Appendix  1 shows  that  if  K+(QQ),  K"(Qq)  are  the 
stabilizing  and  destabilizing  solution  of  the  (ARE)  (3.  57) 

A'K  + KA  + Qq  = KBB'K 

and  K+(Q0)  / K (QQ),  a slight  perturbation  of  QQ  suffices  to  make  the 
strict  inequality,  between  K+,  K~  hold.  Namely  let  Q = + € I. 

It  is  easy  to  show  that  K+(Q^),  K (Q^)  exist  and 

K+(Qj)  > K'(Qj) 

for  any  t > 0.  It  is  in  this  sense  that  we  can  make  the  statement  that 
Bellman's  algorithm  converges  for  almost  any  Riccati  equation  that  has  a 


solution. 
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IV.  STABILITY  OF  INTERCONNECTED  SYSTEMS 


1.  Introduction 

When  confronted  with  the  task  of  analyzing  a complicated  system, 
one  can  usually  achieve  a better  understanding  by  considering  it  as  an 
interconnection  of  simple  subsystems.  This  is  perhaps  a natural  way  of 
understanding  a man-made  system  since  the  engineering  design  process 
is  mostly  one  of  interconnecting  components  to  achieve  a system  with 
desired  behavior  [1],  On  the  other  hand,  it  is  usually  not  obvious  how 
the  behavior  of  a collection  of  subsystems  gives  rise  to  a qualitatively 
different  behavior  of  the  overall  system.  The  fact  that  we  have  little 
intuition  about  the  behavior  of  an  arbitrary  interconnection  of  two  systems 
might  lead  us  to  believe  that  it  is  impossible  in  general  to  study  large 
systems  by  first  analyzing  their  subsystems  and  then  somehow  putting 
the  pieces  together  into  information  about  the  overall  system.  A major  task 
of  large  scale  system  theory  is  therefore  to  respond  to  the  above  objections 
by  identifying  classes  of  systems  for  which  interesting  questions  can  be 
answered  by  studying  their  subsystems  and  their  interconnections  on  a 
separate  basis. 

Part  of  Section  II  dealt  with  such  an  approach  to  the  stability  of 
int*  rconnected  systems,  the  main  step  of  which  was  in  postulating  inter- 
connection .constraints  that  make  possible  the  formation  of  a system 
Lyapunov  function  from  the  subsystem  ones.  The  postulated  interconnection 
constraints  were  to  a great  extent  ad  hoc,  since  it  is  har  1 to  find  any 
reason  for  postulating  them  except  in  order  to  make  the  desired  results  go 
through.  In  this  Section  we  examine  a more  natural  approach  to  the  study 
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of  stability  of  interconnected  systems  due  to  J.  C.  Willems  [2],  [3],  [4], 
the  dissipative  system  approach.  We  have  called  this  approach  natural 
because  the  fundamental  definition,  that  of  a dissipative  system, 
formalizes  the  notion  of  a lossy  physical  system  such  as  a passive 
electric  network.  It  is  thus  expected  that  a large  class  of  systems  will 
satisfy  the  conditions  for  dissipativeness.  The  theory  applies  nicely  to 
large  systems:  A system  will  be  dissipative  if  it  is  the  interconnection 
of  dissipative  subsystems  and  thus  the  property  of  dissipativeness  is 
transmitted  from  lower  hierarchical  levels  to  higher  ones. 

In  2 we  consider  the  dissipative  ness  of  linear  systems  and 
that  of  their  interconnection.  It  is  conceivable  though  that  not  all  sub- 
systems forming  an  interconnected  system  will  be  dissipative.  We  thus 
address  in  3 the  problem  of  transforming  such  subsystems  to  dissipative 
ones  through  linear  feedback  on  the  subsystem  state  only.  This  process 
is  referred  to  as  decentralized  stabilization  and  is  reasonable  when  it  is 
very  costly  to  observe  the  state  of  a subsystem  for  the  purpose  of 
controlling  a different,  remotely  located  subsystem.  The  design  of  such 
decentralized  controllers  would  involve  an  awkward  search  procedure. 

As  an  alternative  we  propose  in  3.  2 a class  of  linear  controllers  motivated 
from  linear  quadratic  game  theory.  These  controllers  can  be  obtained  by 
a straightforward  numerical  procedure  based  on  the  Bellman  algorithm 
(hence  referred  to  as  B-algorithm)  for  solving  a Riccati  equation  that  was 
presented  in  Section  III,  Prop.  3.  59.  The  justification  of  these  techniques 
is  given  in  4.  3.  An  argument  to  the  effect  that  the  proposed  class  of 
linear  controllers  is  sufficient  for  stabilization  purposes  is  given  in  4.4. 
Finally  the  introduction  of  subsystem  state  observers  is  discussed  in  4.  5. 
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2.  Dissipative  Interconnected  Systems 
2.  1 Preliminaries 

A dynamical  system  in  state -space  form  is  defined  [3]  by 
Z = fU,  Y,  X,  0,  r}  where 

U --  input  alphabet 

- input  space;  a class  of  functions  mapping  R+  into  U 
Y = output  alphabet 

- output  space,  analogous  to  °U 
X = state  space 

0 = state  transition  function 

r = readout  function 

2 (*) 

The  state  transition  function  0 : R+  xX  x ^ - X satisfies  the  usual 
state  transition  axiom^ 

U)  0(tq>  x0’  = x0 

(ii)  If  Uj(t)=u2(t)  for  then  <*(tp  tQ,  x,  iij ) = 0(tp  tQ,  x,  u2) 

(iii)  0(^2*  0(^1*  x>  ^ y ^ ® 2 9 toy  x>  ^ 

2 

The  read-out  function  r:XyUxR+-Y  determines  the  input- 
output  structure  of  the  system.  It  is  required  that  r(0(t,  tQ,  x,  u),  u(t),  t,  tQ) 
defined  for  t s t^,  be  the  restriction  to  of  an  element  of  ^ 

denoted  by  y(x,  t^,u). 

In  order  to  introduce  the  notion  of  a dissipative  system  let  us 
assume  that  in  addition  to  a dynamical  system  E we  are  given  two  scalar 

R+  = 2 ^ 


functions  S :XxR-*R 


and  w : Ux  YxR  -*R.  The  function  S(x,  t) 


has  the  interpretation  of  "energy"  content  of  state  x at  time  t and  is 
called  storage  function.  The  function  w(u,  v,  t)  has  the  interpretation  of 
power  supplied  to  the  system  by  the  external  world  and  is  thus  called  the 
supply  rate.  The  supply  rate  depends  on  the  instantaneous  values  of  the 
system's  input  and  output  but  not  on  the  state.  This  definition  reflects  the 
notion  that  energy  can  only  be  transferred  to  a system  through  its  external 
terminals  and  the  transfer  is  not  affected  by  the  inner  workings  of  the 
system,  i.  e.  its  state.  Consider  now  a transition  of  £ from  (Xq,  t^) 
to  (xj,t,)  under  the  influence  of  an  input  u(t).  An  amount 

w(u(t),  y(t),  t)  dt 


of  energy  is  externally  supplied  to  the  system  during  its  transition.  If 
the  inequality 


S(xr 


V 


.t 


w(u(t),  y(t),t)  dt  a S(x1,t1) 


(4.  1) 


holds,  it  means  that  energy  was  lost  in  the  transition  since  the  storage  at 
the  new  state  is  less  than  that  at  the  old  state  plus  the  externally  supplied 
energy.  A system  {£,w,  S]  is  called  dissipative  if  (4.  1)  is  satisfied  for 
for  every  possible  transition  and  in  addition  the  storage  function  is 
nonnegative.  The  Dissipation  Inequality  (DI  for  short)  (4.  1)  is  invariant 
under  addition  of  a constant  to  S and  hence  the  nonnegativity  of  S can 
be  satisfied  if  and  only  if  S has  a finite  lower  bound. 

The  relation  of  dissipative  systems  to  stability  hinges  on  the  fact 
that  if  there  is  no  external  supply  to  the  system,  it  will  dissipate  its 


stored  energy.  In  terms  of  the  previous  definitions,  if  an  input  u(t)  is 
applied  such  that  w(u(t),y(t),t)sO  the  (DI)  implies  that  S(x(t),t)  will 
be  decreasing  in  t and  can  serve  as  a Lyapunov  function  from  which 
stability  results  will  follow  if  S satisfies  the  requirements  of  any 
Lyapunov  stability  theorem. 

One  may  wonder  what  is  the  advantage  of  checking  the  rather 
involved  conditions  for  dissipativeness  if  one  is  only  interested  about  the 
stability  of  a system  for  a certain  class  of  inputs.  It  should  be  pointed 
out  first  that  checking  the  dissipative  ness  condition  does  not  involve 
choosing  both  a supply  rate  w and  a storage  function  S.  For  a given 
supply  rate  w,  it  can  be  shown  that  if  the  nonnegative  expression 


Sa(xO>V  = SUP  - J w(t)  dt  (4.2) 

*1*0  *0 
x0~* 


is  defined,  i.  e.  Sa(Xg,  tg)<  « Vxg,tg,  Sa  can  serve  as  a storage 

function.  The  sup  in  (2)  is  taken  over  all  transitions  of  E in  (tg,tj) 

starting  at  Xq  at  tg,  without  any  restriction  on  tj  or  the  input  used  in 

(* ) 

(tg,tj).  In  fact  it  is  shown  in  [3]  that  Sa  is  the  smallest 

possible  storage  function.  This  result  shifts  the  burden  of  stability 

proofs  to  the  choice  of  an  external  supply  function  w,  a task  about  which 

one  might  have  greater  intuition  than  in  choosing  a Lyapunov  function. 

Perhaps  the  most  important  feature  of  the  theory  is  the  ease  it  can 

be  applied  to  an  interconnection  of  dissipative  systems  [2].  Let  us  assume 

that  we  are  given  a family  of  dynamical  systems  E.  = fU.,'?/.,  Y.,  X.. 

i r r i’  v r 

^i’  ri^  * ^ The  systems  are  then  interconnected  in  some  way  to  form  a 
new  system  with  inputs  {U,?/j  and  outputs  {Y,^}.  A memoryless 


If  S s 0 satisfies  (4.  1),  then  S 2Sa- 


. 


interconnection  can  be  visualized  as 


Figure  1 


i.  e.  as  a function  G : Ux  Y,  x...  Y -•  YxU,  X ...  X U . For  the 

l n x n 

system  in  Fig.  1 to  make  sense  one  must  somehow  verify  that  an  input 
u gives  rise  to  unique  "outputs"  u.  € y^  € 3^,  y € 3^.  This 
property  may  be  quite  hard  to  verify  for  a general  interconnection,  but 
it  obviously  holds  for  the  type  of  subsystems  to  be  considered  here.  So 
let  us  assume  that  the  system  in  Fig.  1 is  well  defined  and  denote  it  by 
X^i  2^  |G.  Assume  furthermore  that  every  is  dissipative  with 
respect  to  a supply  rate  w^  and  a storage  function  S^,  i £ I.  The 
dissipativeness  property  is  transferred  to  the  interconnected  system: 
Namely  the  system  X.^j  X . | G is  dissipative  with  respect  to  the  supply 
rate 

w(u,y,t)  d=f  £ w (u  y t)  (' 

i€I  111 


and  the  storage  function 


The  proof  of  this  fact  is  immediate  [3,  Prop.  HI-1].  It  should  be  noted 
that  for  the  definition  (4.  3)  to  make  sense  it  must  be  possible  to  express 
the  u^,  y\ 's  in  terms  of  u,  a fact  that  is  guaranteed  by  the  assumed  well 
posedness  of  the  problem  and  the  memoryless  property  of  the  inter- 
connections. 

The  above  remarks  provide  the  following  sufficient  conditions  for 
the  stability  of  an  interconnected  system.  Let  us  assume  that 

w(u,y,  t)  = £ w(u.,y.,t)  s 0 

i€l 

Then  S = ES^  is  a Lyapunov  function  for  X.^j  E.  | G since  it  decreases 
along  the  trajectories  and  stability  results  will  follow  if  E.^S^  has 
further  properties  for  Lyapunov  theorems  to  apply. 

2.  2 Stability  Conditions  for  Systems  with  Quadratic  Supply  Rates 

The  inequality 

w(u,y,  t)  s 0 (4.5) 

guarantees  that  the  storage  function  S(t)  of  a dissipative  system  decreases 
during  the  application  of  the  input  u.  Furthermore,  if  the  storage  function 
is  positive  definite,  that  is  0j(  |(x|| ) £ S(x,  t)  s 0^(  |jx|| ) where  0.  are 
scalar  positive  definite  functions,  0(®)  = ®,  inequality  (4.  5)  guarantees  that 
II xt  ]|  is  bounded.  Furthermore  if  w(u,  y,  t)  s - 03(  ||x|| ),  global  asymptotic 
stability  follows.  In  particular  if  a system  is  dissipative  w.  r.t. 

w(u,  y)  = u'u  - p y'y  p > 0 (4.6) 


and  Sa(x,  t)  is  positive  definite,  the  bound  u'u  y'y  guarantees  bounded 


trajectories,  and  u'u  s p y'y  - 0^ ( [| x |j ) asymptotically  stable  ones. 

The  study  of  linear  systems  with  quadratic  supply  rates  has 
received  a good  amount  of  attention.  Consider  the  system 


x = Ax  + Bu  | 

y = cx  j 

The  system  will  be  dissipative  with  respect  to  the  supply  rate  w of 
(4.6)  if  and  only  if 


(4.7) 


Sa(x0’V  = sup 


X<V~X0  fco 


f1 

- J (u'u  - 


px'C'Cx)  dt 


*1  St0 


exists  for  all  Xq.,  t^.  Due  to  the  time  invariance  of  (4.  7)  it  suffices  to 
consider  t^  = 0 and  we  write  Sa(Xg)  instead  of  Sa(xQ,tg).  Furthermore, 


it  can  be  easily  shown  that 

t. 


sup 


f 


X(0)-XQ  0 


w(t)  dt  ^ sup 


r 


w(t)  dt 


x(0)=xQ  0 


for  tj  s 1 2 by  noticing  that  a null  input  u(t)  = 0 for  t p 
concatenated  to  any  input  u(t)  for  t 6 [0 , t j ) will  result  in 


r 


w(t)  dt  s 


-f 


w(t). 


Hence 


- Sa(xQ) 


inf 


x(0)=x 


0 


00 

{ 


{ - p x'C'Cx  + u'u]  dt 


(4.8) 


. .......  


Lifrt 


■ i 


•J 


i 


' j 

* s 


i 
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Theorem  A.  2 of  Appendix  1 shows  that 
Sa(x0)  = ‘ X0K+X0 


(4.9) 


exists  (and  hence  (4.  7)  is  dissipative)  if  and  only  if  the  following  (FDI) 


holds  for  oj  € R 


I - p B'(  -jouI-A')  ”^C  'C(ju)I- A)  ~ 1 B a 0 


(4.  10) 


Now  Sa(x)  is  a positive  definite  function  and  the  additional  condition 

w = u'u  - p x'C'Cx  £ 0 guarantees  the  boundedness  of  the  trajectories  of 

(4.  7).  Global  asymptotical  stability  follows  if  u'u  ^p  x'C'Cx  - 0(x).  ^ 

A further  application  of  the  above  is  to  the  stability  of  interconnected 

systems.  Consider  an  autonomous  system  of  the  form 

xi  = Aixi  + Bihi(xi, . . . ,xN;t)  i=l,...,N  (4.11) 

n. 

where  x.  € R \ B.  is  a n.  x m.  full  rank  matrix  and 
1 ’ i , , „ i i ^ 

n^  + . . . +n  m. 

h^(Xj, . . . , xn;  t)  :R  xR-*R  The  pairs  (A.,  B.)  are  controllable 

V i.  We  can  view  this  system  as  an  interconnection  of  N input-output 


subsystems 


x.  = A.x.  + Bu. 
i ii  l 


(4.  12) 


with  the  interconnection  constraint 


ui  = hi(yi>--->yN;t) 

If  each  input-output  system  in  (4.  12)  is  dissipative  with  respect  to  a supply 
function 


w.(u.,y.)  = u'u.  -p.x'.x. 


(4.  13) 


the  interconnected  system  in  (11)  is  also  dissipative  in  the  supply  rate 

>Js 

The  weaker  condition  u'u  <;  (p -e)x'C'Cx  guarantees  gl.  as.  stability.  See 
[6,  Sect.  33]. 


(4.  14) 


'(t)  = S {u.'u.  - p.x.'x.J 

•Tj  11  ill 


The  supply  rate  (4.  14)  is  denoted  as  w(t)  instead  of  w(u,  y,  t)  since  the 
system  is  autonomous.  In  view  of  the  remarks  at  the  beginning  of  this 
section,  boundedness  of  the  trajectories  interconnected  system  (4.  11)  is 


guaranteed  if 


IN 

w(t)  = 53  {u.'u.  - p.x.'x.}  S o 

iTl  11  ill 


or  equivalently 


A N 

hi  (xl",-»xN;t)hi(xi>--.,xN;t)  s ^ p x-xi 


(4.  15) 


Inequality  (4.  15)  is  a condition  on  the  interconnection  of  the  system  (4.  11). 

It  is  a distinct  improvement  on  the  conditions  that  could  be  obtained  without 

the  application  of  the  facts  about  the  interconnection  of  dissipative  systems.. 

Each  subsystem  of  (4.  12)  would  be  stable  if  u.'u  s p x'x  or 

ii  l i i 


hJ(xi, . . . , Xj^;  tjhftxp  . . . , Xj^;  t)  ^ p^x^x. 


(4.  16) 


If  (4.  16)  is  satisfied  for  all  i the  interconnected  system  (4.  11)  is  stable. 
This  is  a much  stronger  stability  condition  than  (4.  15).  In  fact  (4.  16)  is 
very  restrictive  since  it  implies  that  h.fxj, . . . ,xN)  depends  on 
Xj,  j / i in  a very  weak  manner. 

Similar  remarks  can  be  trade  for  the  interconnection  of  systems 
which  are  dissipative  with  respect  to  different  supply  rates.  For  the 
quadratic  rate 


the  minimal  nonnegative  storage  function  can  be  obtained  as 


1 

= 7 sup 

*1*0 


2u'Py  dt 


which  becomes,  by  making  the  same  transformations  as  in  the  derivation 
of  (4.  8),  and  setting  y = x 


Sa^x0) 


l 


2u  'Px  dt 


According  to  Theorem  A.  1?  the  infimum  exists  if  and  only  if  the  (ARE) 


A'K  + KA  = (KB  + P'MB'K  + P) 


has  a negative  semi-definite  solution.  If  it  possess  s a negative  definite 
solution  K+, 

Sa<*0>  - - I *0K+X0 

and  if  in  addition  K+  is  feedback  stabilizing,  the  infimum  is  a minimum 
and  is  attained  by  the  input  u = - (B'K+  + P)x. 

Applying  these  results  to  the  interconnected  system  (4.  11)  we  obtain 
an  analog  to  stability  condition  (4.  15)*  as 
N 

53  h.(xj xN;t)P.x.sO  (4.17) 


under  the  assumption  that  the  subsystems  in  (4.  12)  are  dissipative  with 
respect  to  the  supply  rates  w. (u.,y.)  = u.P.x.. 


The  m.  X n.  weighting 
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matrices  Pj  are  unrestricted,  as  long  as  the  subsystems  are  dissipative 
and  should  be  chosen  so  as  to  facilitate  the  verification  of  (4.  17). 

In  the  remainder  of  this  section  we  will  be  mainly  concerned  with 
supply  rates  of  the  form  (4.  13),  i.  e.  w(u,  y)  = u'u  - p y'y.  Most  of  what 
will  be  said  about  them  will  apply  to  inner -product  supply  rates 
w(u;  y)  = u'Py  since  the  verification  of  dissipativeness  in  both  cases 
involves  algebraic  Riccati  equations.  Furthermore  an  interconnection 
constraint  involving  norms  seems  more  plausible  than  one  involving  inner 
products. 

Given  that  the  stability  condition  (4.  15)  becomes  weaker  with 
increasing  we  are  interested  in  finding  the  largest  possible  values 
for  Pj,  i = 1,  . . . , N which  will  then  provide  the  best  possible  stability 
condition  of  the  form  (4.  15).  An  equivalent  problem  is  to  determine  the 
range  of  p such  that  the  system 

x = Ax  + Bu 

is  dissipative  with  respect  to  the  supply  rate  w(u,  y)  = u'u  - px'x.  Again 
this  is  equivalent  to  finding  the  range  of  p for  which  the  (ARE) 

A'K  + KA  - pi  = KBB'K  (4.  18) 

has  a negative  definite  solution,  or  finding  those  p for  which  the  Frequency 
Domain  Inequality  (FDI) 

I - p B'(-jwI  - A')'1  (jul  - A)'1  B :»  0 (4.  19) 

holds  for  all  to.  Numerical  methods  based  on  either  the  (ARE)(4.  18)  or 
the  (FDI)  (4.  19)  are  possible  in  view  of  the  fact  that  the  range  of  p is  a 
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closed  interval  [0 , ]-  A good  estimate  for  can  be  obtained  by  a 

half-interval  search  procedure.  If  p^  is  known  to  lie  in  an  uncertainty 
interval  [a,  b]  a search  at  the  half  point  h = a+b/2  will  show  whether 
Pq  is  in  the  right  or  the  left  half  of  the  interval  according  to  whether  the 
(ARE)  (4.  18)  has  or  has  not  a negative  definite  solution.  To  ascertain 
the  existence  of  a solution  of  (4.  19)  the  B-algorithm  presented  in  Section  III 
can  be  used.  We  show  later  that  it  is  applicable.  We  first  present  the 
entire  algorithm  for  the  determination  of  p^  up  to  a desired  accuracy  e. 


Algorithm  1 


Step  1 . Let  a = 0 and  b = M,  M an  arbitrary  positive  number. 

Step  2 . Use  the  B-algorithms  to  determine  whether  (4.  18)  has  a 
nondefinite  solution  for  p = b.  If  it  does,  let  a *-b,  b — 2b  and  repeat  this 
step.  If  it  does  not,  proceed  to  Step  3. 

Step  3 . Upon  exiting  Step  2,  we  know  that  p^  € [a,  b].  Let 
h - a+b/2  and  use  the  B-algorithm  to  see  if  (4.  18)  has  a nondefinite 
solution  for  p = h.  If  it  does  set  a <-h  and  go  to  Step  4.  If  not,  set 
b *-h  and  go  to  Step  4. 

Step  4 . If  |a-b|  < e,  the  desired  accuracy  has  been  achieved  and 
we  stop.  If  not,  repeat  Step  3 with  the  new  values  of  a,  b. 

Proof.  For  a fixed  uu  , inequality  (4.  19)  is  satisfied  for  p 6 [0,  max 
\{  B'(jul  - A')  *(jwl  - A)  *B}]  = [0,p  (to)],  a closed  interval.  The  range  of 
p satisfying  (4.  19)  for  all  w is  H [0,p(u)],  which  is  a closed  inter- 
val,  being  an  intersection  of  closed  sets. 


- 
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The  validity  of  Algorithm  1 depends  on  the  convergence  of  the 
B-algorithm  for  D < pg.  Its  convergence  can  be  proven  as  follows. 
According  to  Proposition  3.  58  we  must  show  that  K*  >K  . According  to 
Theorem  A.  3,  this  is  equivalent  to  showing  that,  for  some  e>0, 


Hp(-ju,ju)  = 
2 


I - pB'(-juI  - A 1 ) ~ 1 ( jul  - A)-1B 
e B«(-jWI  - A')_1(jul  - A)_1B 


Since  H (-jw,  ju)  s 0 the  strict  inequality  holds  for  p<  p , 

P0  0 

e = pg  - p . Hence  the  B-algorithm  converges  for  p <Pg-  If  it  is  applied 
for  p >Pg,  computational  experience  shows  that  instability  will  soon 
occur  thus  showing  that  the  (ARE)  (4.  18)  does  not  have  a nondefinite 
solution. 

Although  Algorithm  1 will  work  no  matter  what  choice  is  made  for 
the  initial  upper  point  M of  the  uncertainty  interval  (Step  1),  a good 
choice  for  M can  be  made  on  the  basis  of  the  (FDI)  (4.  19).  The  maximum 
p satisfying  (4.  19)  for  u = 0 is  an  upper  bound  for  Pg.  The  (FDI)  becomes 
lap  B'(AA')  *B  for  w = 0 and  the  largest  p satisfying  it  is 

p = [max  \{B'(AA')  *B}]  * = M. 


We  are  guaranteed  that  the  (ARE)  (4.  18)  does  not  have  a nondefinite 
solution  for  p = and  thus  pg  is  included  in  [0,Mj].  We  can  therefore 
start  Algorithm  1 at  Step  3 with  a = 0,  b = Mj  . 

Given  that  the  (FDI)  (4.  19)  is  an  algebraic  condition,  one  would  hope 
for  an  analytic  method  for  computing  pg.  Of  course,  a straightforward 
verification  of  the  (FDI)  (4.  19)  is  inefficient  even  for  a given  p.  For 


special  cases  though  it  provides  a superior  way  of  finding  p„.  Rewriting 


the  minimum  of  the  L.  H.S.  is  at  u = 0 and  hence 


i.  e.  the  bound  M.  is  precisely,  p 


We  note  finally  that  an  inspection  of  (4.  19)  shows  that  it  will  definitely 


hold  for  u large  enough.  We  give  here  an  estimate  of  how  large  is 
"large  enough",  an  estimate  that  simplifies  the  task  of  a straightforward 
verification  of  (4.  19).  Consider  the  expression 


when 


which  is  again  Hermitian.  It  can  be  shown  that  0'  is  positive  definite  for 
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* 

and  negative  definite  for  u < - « 0.  Therefore  0 is  increasing  for 

|u|  >Uq  and  equivalently,  the  L.  H.S.  of  (4.20)  decreases  for  |u|  >Uq- 
In  verifying  that  (4.  20)  or  (4.  19)  holds  for  a fixed  p one  has  to  check 
(4.20)  for  |uj  < Uq  rather  than  all  u € R.  Even  so,  this  is  a much  less 
efficient  procedure  than  Algorithm  1. 


3.  Decentralized  Stabilization 


3.  1 Linear  Controllers 

We  consider  now  the  interconnected  control  system 


x. 

l 


A.x.  + B.u.  + 
ii  ii 


hi(xP 


(4.21) 


n.  m.  n +.  . . +n  n. 

with  x.  € R % u.  6 R 1 and  h.  : R N x R - R 1 . The 

difference  between  this  system  and  system  (4.  11)  is  that  the  latter  is 


* 

Proof.  Consider  first  a Hermitian  matrix  M = P + jQ,  P = P'  and 
Q = - Q'.  We  get  by  direct  computation  that 


(x1  - jy')M(x  + jy) 


c :>  0 


The  positive  definiteness  of  M is  equivalent  to  that  of 
Note  that 


/I  0 \ / P Q\  /I  -P_1Q'\  IP  0 \ 

\-QP_1  I / \Q'  P/  \0  I / \0  P-QP_1Q'/ 

Thus  M > 0 <=>  t (i)  P>0  (ii)  P-QP_1Q'>0.  For  M = 2ul  + J(A-A') 

= <2> 1 and  w > 0 we  get  that  0' > 0 2ul  - ((A-A')(A'-A)/2w)  > 0 or 

2 

4u  I > (A-A')(A' -A).  This  can  hold  if  and  only  if  u > = l/2fmax  X.[(A-A') 

1 /2 

(A'-A)]J  iO.  It  can  be  similarly  shown  that  01  < 0 for  u)  < -Uq  s 0. 
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autonomous,  while  (4.21)  admits  inputs  u.,  i = 1, . . . , N.  It  is  then 
possible  to  apply  feedback  to  (4.  21)  in  order  to  modify  it  in  some  desirable 
way.  We  assume  here  that  a feedback  controller  on  u.  can  be  a function 
of  x.  or  of  an  observable  output  y.  = y^(x^)  °f  the  i-th  subsystem.  In 
other  words,  it  is  too  costly  to  implement  a controller  for  the  i-th  sub- 
system who  can  observe  the  state  of  any  subsystem  different  from  i. 

An  essential  feature  of  any  controller  is  that  it  must  lead  to  a stable 

system.  In  this  and  the  next  section  we  investigate  the  question  of 

decentralized  stabilization  of  system  (4.  21),  i.  e.  the  design  of  controllers 

u.  = u.(-y.(x.))  which  stabilize  (4.  21)  in  the  presence  of  the  interconnection 

terms  h..  Time -invariant  controllers  linear  in  the  subsystem  state  are 
l 

considered.  We  will  assume  that  either  the  entire  state  x^,  i = 1,  . . . , N 
or  subsystem  outputs  y.  = C^x.  are  available. 

In  the  previous  section  we  showed  that  a bound  on  the  interconnections 
of  the  form 


N N 

22  h|(xl » • • • > xn>  *)kj(xj>  • • • > x^j5  s X Pjxjxj 
i=l  i=  1 


(4.  15) 


guarantees  the  stability  of  an  interconnected  system  provided  the  sub- 
systems are  dissipative  with  respect  to  the  supply  rate  w.(u.,  y^)  = 

= u.'u.  - o.y.'y..  This  result  might  not  be  applicable  to  (4.  21)  if  A.  is  not 
l l ri  l l i 

8 table  for  some  i € 1, . . . , N and  u..  = 0.  The  i-th  subsystem  will  not 
be  dissipative  for  any  pi  . On  the  other  hand,  if  (A.,  B^)  is  a controllable 
pair,  a feedback  u‘.  = P.jc.  can  be  chosen  to  make 


A.  + B.P. 
l ii 


stable.  The  system 


‘i  * Ap*i + vi  * *1**1 


(4.22) 


will  be  dissipative  with  respect  to  w.(v.,  y^)  = v/v.  - p^x/x^  provided  the 


(FDI) 


(cj  - p )I  + ApAp  + ju(Ap  - Ap)  * 0 

is  satisfied  for  all  w . Since  the  (FDI)  holds  for  p.  = 0 it  will  hold  for 
some  sufficiently  small  o-  > 0*  It  follows  therefore  that  by  stabilizing 
each  subsystem  we  can  guarantee  its  dissipativeness  for  a small  enough 
Pi  and  hence  we  can  guarantee  the  stability  of  the  overall  subsystem  for 
interconnections  small  enough  to  satisfy  (4.  15)  for  the  above  pJs. 

It  is  perhaps  natural  to  conjecture  that  by  placing  the  poles  of  a 
subsystem  far  into  the  left  half  plane  it  can  be  made  dissipative  with 
respect  to  any  supply  rate  w(u,  y)  = u'u  - p y'y  i.e.,  for  any  p > 0.  This 
conjecture  is  false  as  can  be  seen  in  the  following  example. 

Consider  a system 


x = (A  + BP)  x + u 


y = x 


(4.23) 


where  x,  u € R , Bisa  n y m matrix,  Pa  m x n one  and  (A,  B) 
is  controllable.  P is  to  be  chosen  so  that  Re  \(A  + BP)  < 0 and  (4.23) 
is  dissipative  with  respect  to  w(u,  y)  = u'u  - px'x  for  a given  p.  We  are 
interested  in  establishing  the  0 for  which  the  choice  of  such  a P is 
possible.  In  particular  consider 


iMMN 
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(4.  24) 


For  this  example, 


A + BP 


and  it  is  stable  provided  p,  q < 0.  Furthermore 


(B,  AB) 


so  that  (A,  B)  is  controllable.  For  a given  p the  system  (4.  24)  is 
dissipative  provided 


(u2  - p)I  + (A  + BP)'(A  + BP)  + jw[(A  + BP)  - (A  + BP)']  a 0 


for  all  u or,  after  some  algebra,  if 


p2  + (u)2  - p ) 
pq  - j<j(l  - p) 


pq  + jw(l  - p) 
q2  + 1 + (u2  - p ) 


* 0 


(4.25) 


for  all  u.  The  Sylvester  criterion  applied  to  (4.  25)  results  to  the 
inequalities 
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[p2  +(«2  - p)]  [q2  + 1 + (oj2  - p)]  - [p2q2  + <J2(1  - p)2]  > 0 (4.27) 


for  all  w € H.  Inequalities  (4.  26)  holds  for  all  u if  and  only  if  p >p  . 

Now,  for  a fixed  p a straightforward  computation  shows  that  the  minimum 

2 2 

of  the  L.  H.S.  of  (4.  27)  occurs  at  either  u>q  = p - p - q /2  if 

2 2 2 
p - p - q / 2 > 0 or  at  Wq  = 0 if  p - p - q /2s0.  In  the  first  case, 

2 

the  L.  H.S.  of  (4.27)  for  Uq  becomes 


- (p+v  N 


LHS  = - p - \ p + / + p^(l  - p)  + 2pp 


(4.  28) 


and  in  the  second  case,  w = 0 


LH5  = p2(l  - p)  + p2  - p - p2q2 


(4.29) 


In  either  case  it  can  be  checked  that  for  1 - p < 0 both  expressions  are 
ne  gative. 

This  example  shows  that  no  matter  what  linear  feedback  is  appliedto  (4.24), 
it  can't  be  made  dissipative  for  p > 1.  The  result  is  intuitively  plausible. 
Consider  again 


o ■ c :)  CD  • a 


Note  that  for  u = - x_  results  in  x,  = 0 and  thus 
1 2 1 

If  p > 1 the  supply  rate  w(u,  y)  for  u^  = - x^,  u^ 


Xj(0)  = const. 

0 is  negative, 


This  is  obvious  for  (4.  28).  As  far  as  (4.  29)  is  concerned,  its  supremum 

2 2 2 
over  p > p is  at  p = p and  equals  - p q <0. 
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in  fact  w(u,  v)  = (p  - l)xj(O)  - p x^(t)  s (p  - l)xj(O)  for  all  t and  thus 


Sa(xQ)  = sup 

x„-» 


1 


w(t)  dt 


is  unbounded  provided  Xj(0)  / 0.  Thus  (4.  24)  is  not  dissipative  for 
p > 1.  The  catch  is  that  in  (4.  24)  linear  feedback  modifies  only  the  x^ 
equation  but  not  the  Xj  equation  which  can  be  made  unstable  even  when 
w(u,  y)  s 0,  i.  e.  u'u  £ p x'x  provided  p > 1. 

The  above  example  shows  that  the  determination  of  the  possible 
p 's  for  which  a feedback  controller  u = Px  exists  to  make  the  system 
dissipative  in  the  rate  w(u,  y)  = u'u  - p y'y  is  a meaningful  problem, 
i.  e.  , we  are  not  guaranteed  that  for  any  p such  a P exists.  In  fact  it 
can  be  shown  that  the  set 


= j 0 | a a P = P{p)  such  that  the  system  x = (A  + BP)x  + u; 

y = x is  dissipative  in  w = u'u  - o x'x  j 

is  bounded  above  if  Rank  (B)  < n,  n being  the  dimension  of  x.  More 
precisely,  R is  an  interval  of  the  form  [0,pg)  or  [0,pQ]  where 
Pq  = 1.  u.  b.  2ft  . 

Determining  pQ  is  an  important  problem,  as  it  provides  a measure 
of  our  capacity  to  stabilize  a nonlinear  system  with  linear  feedback  control. 
In  the  above  2-dimensional  example  we  showed  by  using  the  frequency 
domain  approach  that  Pq  ^ The  determination  of  the  exact  value  of 
Pq  by  a frequency  domain  method  would  lead  to  a harder  problem. 

In  general,  for  a given  p one  would  like  to  have  an  algorithm  for 
determining  a n x ni  matrix  P = P(p)  such  that 
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(i)  Re  \(A  + BP)  = Re  \(Ap)  < 0 

(ii)  I - p( -jul  - Ap)  ^(jul  - Ap)  1 s 0 V u 


(4.30) 


if  such  a P exists.  In  the  next  section  we  will  provide  such  an  algorithm, 
the  motivation  of  which  comes  from  the  study  of  infinite  duration  games. 

The  procedure  will  generate  control  matrices  P by  solving  Riccati  type 
equations  by  a numerical  procedure  similar  to  Algorithm  1.  The  convergence 
of  the  procedure  will  also  be  examined. 


3.  2 Game  Theoretic  Stabilization 
Let  us  consider  the  system 


x = Ax  + Bu  + v (4.  31) 

« 

We  want  to  design  a linear  controller  u = Px  such  that 


x = (A  + BP)x  + v 

is  dissipative  for  the  rate  v'v  - p x'x,  or  equivalently 


sa<xo}  = 


inf 

v 


CO 

l 


{-  p 


x'x  + v'v} 


dt  < » 


(4.32) 


x = (A  + BP)x  + v x{0)  = Xq 


If  such  a controller  exists,  then  the  expression 


inf  Sa(xQ) 
u-Px 


sup 

u-Px 


inf 

v 


^ {-  p x'x  + v'v]  dt 


(4.33) 


is  finite.  The  form  of  (4.  33)  motivates  us  to  consider  the  following  class 
of  problems  in  the  positive  parameters  T|,  0: 


00 

= sup  inf  J 
u v V 


(-  x'x  - 0 u'u  + n v'v)  dt 


(4.  34 


x = Ax  + Bu  + v ; x(0)  = xr 


The  understanding  is  that  the  infv,  3UPU  are  taken  over  non- 
dynamic closed  loop  strategies.  In  (4.  34)  the  constraint  u = Px 
for  some  P is  relaxed  and  the  term  - Q u'u  has  been  added  to  the 
integrand.  These  changes  have  been  made  in  the  expectation  that  for  a 
significant  range  of  T}  J will  be  finite  and  the  sup^  will  be  a minimum 
at  some  linear  control  u = Mx.  If  this  is  so,  the  following  relation 
between  (4.  34)  and  (4.  33)  is  evident 


inf  Sa(xQ)  ~ ~ P SUP  inf 
u=Px  u=Px  v 


- x'x  + — v'v  dt 
P 


^ - p s up 

u=Px 


inf 

v 


J 


x'x  - 0 u'u  + — v'v  dt 
P 


£ - p sup  inf 

u v 


jf  - x'x  - 0 u'u  + 


1 

0 


v'v 


dt  < ® 


The  finiteness  of  J in  (4.  34)  for  some  rj , 0 implies  that 


inf  Sa(xn)  <® 
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for  p = l/t)  and  in  fact  the  system  x = (A  + BM)x  + v will  be 
dissipative  in  the  rate  w = v'v  - l/r\  x'x  and  hence  the  condition 
f'(x,  t)f(x,  t)  ^ i/n  x'x  will  guarantee  stability.  In  the  sequel  we  will 
provide  an  algebraic  criterion  for  determining  the  finiteness  of  (4.  34) 
for  given  parameters  as  well  as  a procedure  that  will  generate  the 
appropriate  linear  controller  u = Mx. 

The  price  one  pays  for  the  above  simplification  is  a conservativeness 
in  the  estimate  of  the  range  of  p , i.e.  of  Jtf.  Indeed,  problem  (3.  34) 
corresponds  to  determining  the  dissipativeness  of  a system  in  the  supply 
rate  v'v  - p(x'x  + p g u'u)  which  is  smaller  than  the  original  rate 
v'v  - p x'x.  For  a fixed  0,  the  range  of  p (equivalently  of  r))  for  which 
(4.  34)  has  a finite  solution  is  strictly  included  in  This  conservative  - 

ness  can  be  removed  by  taking  0 be  small  enough.  We  will  show  in  the 
next  section  that  if  p € >5?  and  p < p^  = 1.  u.  b.  then  for  all  0 < 0 (p  ) 
and  r)  = l/p  the  inf-sup  in  (4.  34)  will  be  finite.  Thus  if  a stabilizing 
control  exists,  it  can  be  found  by  solving  (4.  34)  for  0 sufficiently  small. 

For  ease  of  exposition  we  assume  in  the  remainder  of  this  section  that 
0=1  and  return  to  the  case  0 —0  in  3.4. 

We  would  like  to  derive  conditions  for  the  existence  of 

OO 

J(Xq)  = inf  sup  J*  x'C'Cx  + u'u  - r)  v'v  dt 
u v 0 

(4.35) 

x = Ax  + Bu  + Dv  x(0)  = Xq 

I 

where  (A,  C)  is  observable  and  (A,  B),  (A,  D)  controllable.  This  is  a more 
general  version  of  (4.  34).  The  inf-sup  problem  (4.  35)  is  intimately 
related  to  the  game  with  value  functional 


MMM 


where  u is  the  minimizing  player  and  v the  maximizing  player.  This 
game  is  treated  in  detail  in  the  next  chapter.  Here  we  present  the 
results  relevant  to  the  inf-sup  problem  (4.  35);  proofs  which  are  given  in 
the  next  chapter  are  omitted. 

PROPOSITION  4.  1.  A number  p is  the  value  of  a game  if 

inf  sup  J = sup  inf  J = p.  Consider  the  game  (4.  36).  Then 
u rv  rw  u r 

(a)  A value  of  (4.  36)  in  closed  loop  strategies  exists  if  and  only  if 
the  Algebraic  Riccati  Game  equation  (ARG) 


A'K  + KA+C'C  = K(BB'  - \/r\  DD')K  (4.37) 

has  a real  positive  definite  solution. 

(b)  If  (4.  37)  has  a positive  definite  solution  for  some  q it  has  a 

-f  -f. 

smallest  positive  definite  solution  denoted  by  K = K (n).  The  value  of 
(4.  36)  is 

y(x0)  = *oK+xO 

Furthermore  Re  \(A  - BB'K^  + l/q  DD'K+)  s 0. 

The  inf-sup  problem  (4.  35)  is  finite  if  and  only  if  the  game  (4.  36) 
has  a finite  value. 

Proof.  If  a finite  value  exists  the  inf- sup  is  finite.  However  if  a 
value  does  not  exist  for  some  initial  condition  x^,  it  will  be  shown  in  the 


2- 


next  chapter  that  the  maximizing  player  can  guarantee  an  arbitrarily 

large  value  of  the  value  functional,  and  thus  inf^  supv  £ sup^  inf^ 

for  every  M,  i.  e.  the  inf-sup  does  not  exist.  Q.  E.  D. 

A further  important  property  of  the  inf-sup  (4.  35)  is  that  it  is 
actually  a min-sup,  the  min  occurring  at  the  controller  u = - B'K+x. 

PROPOSITION  4.  2.  Assume  that  K+  = K+(r|)  exists.  Then  the 
controller  u = - B'K^x  attains  the  inf-sup  for  problem  (4.  35). 

Proof.  The  value  of  the  inf-sup  is,  according  to  Proposition  4.  1 
x^K+Xq.  When  u = - B'K+x  we  have 

00 

J = - inf  I -x'(C'C  + K+BB'K+)x  + r?  v'v  dt 
v T) 

x = (A  - BB'K+)x  + Dv  x(0)  = xQ 

The  corresponding  Riccati  equation  is 

(A  - BB'K+)'M  + M(A  - BB'K+)  - (C'C  + K+BB'K+)  = ^ MDD'M 

It  can  be  verified  by  inspection  that  M = - K+  is  a negative  definite 
solution  satisfying 

Re  X(A  - BB'K+  + 1/r)  DD'K+)  <:  0 

It  follows  from  Theorem  A.  1.3  that  the  value  of  the  infimum  is  XqK+Xq  . 

Q.  E.  D. 
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3.  3 Iterative  Solutions  of  the  Game  Riccati  Equation 

We  consider  now  the  problem  of  establishing  the  range  of  r|  for 
which  (4.  35)  is  finite.  This  is  equivalent  to  finding  the  r)'s  for  which 
the  (ARG)  (4.  37)  has  a positive  definite  solution.  As  a first  step  we  will 
transform  the  (ARG)  to  a Riccati  equation  for  which  the  Bellman 
procedure  we  used  in  Algorithm  1 is  again  applicable.  Let  us  assume 
that  K = K'  > 0 is  a solution  of  (4.  37).  Then  M = K is  a solution 
of 

- AM  - MA’  + (BB1  - 1/n  DD')  = MC'CM  (4.38) 

Conversely  if  (4.  38)  has  a solution  M>0,  then  K = M ^ > 0 is  a solution 
of  (4.  37).  The  (ARE)  (4.  38)  is  of  the  type  covered  in  Theorem  4.  3 since 
the  pair  (-A',C()  is  controllable,  (A,  C)  being  observable.  Applying 
now  Theorem  4.  3 the  (ARE)  (4.  38)  we  see  that  since  it  was  shown  to  have 
a solution  M > 0,  it  will  possess  in  addition  a pair  of  real  symmetric 
solutions  M+,M  such  that 

(a)  M~  £ M £ M+  V real  symmetric  solution 

(b)  Re  \(~  A'  - C'CM+)  £ 0 
Re  M-  A'  - CCM'J  * 0 

Thus  if  M > 0 is  a positive  definite  solution,  M+  will  also  be 
positive  definite.  Under  the  conditions  of  Proposition  3.58,  i.  e., 

M+  - M~  >0,  the  B-algorithm  will  converge  to  M+  and  a check  of  the 
positivity  of  the  numerically  computed  M+  suffices  to  determine  whether 
(M+)  ^ is  a positive  definite  solution  to  the  (ARG)  (4.  37). 

The  condition  M+  - M~  >0  is  not  restrictive  in  the  case  where  D 
is  an  n x n identity  matrix,  which  is  exactly  the  case  of  interest,  i.  e.  , 


problem  (4.  34).  The  (ARE)  (4.  38)  becomes 


- AM  - MA'  + (BB'  - 1/n  I)  = MC'CM  (4.39) 

It  is  clear  that  if  (4.  38)  has  a real  symmetric  solution  for  some  rjj  it 
will  have  solutions  for  all  ti  s n j-  In  fact  by  remark  similar  to  those 
used  following  the  (FDI)  (4.  19)  we  conclude  that  (4.  39)  has  solutions  for 
q £ [r)0,  oo ) and  for  any  r|  € (t)q,  ® ) we  have  that 

Re  \(-  A'  - CC'M+)  <0  * 

By  Theorem  5.  3 of  [5]  this  is  equivalent  to  M+  - M >0.  We  thus 
showed  that  the  numerical  procedure  will  converge  for  every  q < iq q . 

The  above  remarks  have  a further  consequence.  Let  us  assume 
that  for  some  r)  / ri0  M+  = M+(q)  > 0.  Since  M+  is  the  maximal  solution 
of  (4.39),  (M+)~*  is  the  minimal  positive  definite  solution  of  (4.  37)  (with 
D = I of  course).  Thus  K+  = (M+)_1.  We  can  rewrite  (4.  39)  as 

(-  A'  - C'CM+)  = (M+f  \a  - BB'K+  + l/n  K+)M+ 

which  shows,  in  view  of  Re  X.(-  A'  - C'CM+)  <0,  that 

T~ 

Proof.  The  (FDI)  corresponding  to  ti0  is 

I + C ( - jgjI  + A')'  1BB'(jwI  + A)_1C'  - l/n0  C(-juI  + A)_1(jwl  + A')-1  * 0 

for  all  u.  We  can  rewrite  the  above  (FDI)  as 

I + C(-jwI  + A')_1BB'(jwI  + A)_1C'  - l/n ! C(-juI  + Aj'^jul  + A')"1 

^ ( l/r|0  - 1/Hj)  C(-jwI  + A)_1(ju)I  + A)"1 

Thus  a\t  r|  ^ the  strict  form  of  the  (FDI)  holds  and  the  result  follows  by 
Theorem  A.  1.2. 


for  T|  / Tin  i.  e.  that  K is  strictly  feedback  stabilizing  for  almost  all  the 


allowed  range  of  t)  . 

The  end  result  of  this  discussion  is  the  reduction  of  the  problem  of 


real  solutions  to 


a question  that  can  be  answered  by  an  iterative  procedure  for  a given  q 


The  determination  of  all  T]'s  for  which  M > 0 is  straightforward  since 
we  know  that  the  q's  lie  in  an  interval  (t|q,®)  or  [Hq,00).  A*1  arbitrarily 

small  interval  [a,  b]  containing  T)q  can  be  found  by  a half-interval  search 
algorithm  similar  to  Algorithm  1. 


Algorithm  2 


Step  1 . Let  a = 0,  b = M,  M arbitrary 


Step  2 . Use  the  B-algorithm  to  see  if  (4.  38)  has  a positive  definite 
r]  = b.  If  not,  a »-  b,  b *-  2b  and  we  repeat  this  step.  Else, 


solution  for 


s a positive  definite 


solution  for  r)  = h.  If  so  set  b *- h and  proceed  to  Step  4.  Else,  set 


<e,  a desired  accuracy,  stop.  Else  repeat 


The  crucial  step  in  both  Algorithms  1 and  2 is  the  iterative  solution 
of  the  (ARE)  for  a sequence  of  p or  T)'s.  For  an  implementation  on  a 
Wang  2200  computer  the  B-algorithm  for  a 10 -dimensional  system 
converges  to  4 significant  digits  in  about  10  iterations,  each  involving  the 
solution  of  a linear  matrix  equation.  These  figures  are  consistent  with 
those  reported  in  [8].  The  convergence  is  quadratic  if  the  initial  control 
happens  to  be  close  to  the  actual  solution  [8],  and  in  practice  2-4  iterations 
would  suffice  for  4 figure  accuracy  if  a good  initial  control  is  chosen. 

This  happens  to  be  the  case  in  Algorithms  1 and  2 where  the  control 
u = - CM+(n)x  would  provide  a good  initial  choice  for  solving  (4.40)  for 
any  ri  j close  to  rj. 

The  numerical  bottleneck  is  in  solving  the  linear  matrix  equation 
A'K  + KA  = - Q.  This  is  equivalent  to  a m = n(n+l)/2  dimensional  linear 
system  where  n is  the  dimension  of  the  state.  To  solve  this  system 
M = m^/3  multiplications  are  needed.  For  ten  such  iterations  the  computer 
time  denoted  to  multiplications  is  approximately 


p being  the  machine  multiplication  time.  For  n = 10  and  a fast  machine, 
p = 20  y 10  ^ the  computer  time  is  T =-  1 secs.  , an  acceptable  time 
figure.  In  view  of  the  n^ -dependence  of  T on  n,  the  time  requirements 
would  be  prohibitive  beyond  n = 30,  in  which  case  computer  time  of  almost 
an  hour  will  be  needed.  Still,  for  system  up  to  n = 20  the  algorithm  can 
be  carried  out  effortlessly  in  a medium  capacity  machine. 


3.4  The  Determination  of  ?Jt 


The  set  introduced  in  Section  3.  2 consists  of  p's  for  which  the 
system 


x = Ax  + Bu  + v 


(4.41) 


can  be  made  dissipative  in  w = v’v  - p x'x  following  the  application  of  a 
linear  feedback  u = Px.  If  p € the  system 

x = Ax  + Bu  + h(x,  t) 


can  be  stabilized  for  h'h  £ p x'x.  The  set  provides  thus  a measure  of 
our  capacity  to  stabilize  a nonlinear  system.  We  apply  here  the  inf-sup 
approach  to  determine  whether  a given  p belongs  to  it  . 

Let  us  consider  the  problem 


J(Xq)  = inf  sup  J p x'x  + 0 u'u  - v'v  dt 
u v 0 


x = Ax  + Bu  + v 


x(0)  = xr 


(4.42) 


We  showed  in  3.2  that  J(xq)  < « for  some  p,0  implies  that  (4.41) 
can  be  made  dissipative  in  w = v'v  - p x'x.  Now,  for  9 small  the 
functional  in  (4.  42)  differs  by  little  from  the  functional  used  in  defining 
Sa(xo)  (4.8).  We  expect  therefore  that  for  small  enough  0 the  finiteness  of 
Sa(Xg)  and  J(xQ)  in  (4.42)  are  equivalent.  This  is  proven  next  in 


LEMMA  4.  1 For  any  p less  than  p^  = 1.  u.  b.  a?  there  exists  a 0(p) 
such  that  the  inf- sup  J(p,0,Xq)  of  (4.  42)  is  finite  for  any  Xq  and  any  0, 

0 s 0 S0(p). 


' 


* 


.■■■■  .11  ■ . ■ . ■ If  — "H 
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Proof.  Let  pj  = p + Pq/2  6 St  . There  exists  a P = P(p  such 
that  the  system 


x = (A  + BP)x  + v 


(4.43) 


is  dissipative  in  w = v'v  - p ^x'x.  Let  m>  0 be  such  that  P'P  < ml. 
For  J(p,0,Xq)  = J(Xq)  in  (4.42)  we  have 


J(Xq)  £ sup  ^ p x'x  + fl  x'P'Px  - v'v  dt 


(4.44) 


x = (A  + BP)x  + v 


Now  pi  + 0 P'P  £ (p  + 0m)I,  and  for  0 < (p  1 - p )/m  - 8 (p  ) we  have 
pi  + 0 P'P  £ pjl.  Hence 


J(xQ)  £ sup 
v 


00 


x'x  - v'v  dt 


x = (A  + BP)x  + v 
which  is  finite  by  our  assumptions  on  p^  and  P. 


Q.  E.  D. 


In  view  of  Lemma  4.  1 and  the  observations  preceding  it,  we  have 
proved 


PROPOSITION  4.  3:  The  inequality  p<p^  holds  if  and  only  if  the 

inf-sup  J(Xq)  in  (4.42)  is  finite  for  some  0 > 0.  This  is  equivalent  to  the 
(ARG) 


-1, 


A'K  + KA  + pi  = K(0  BB'  - I)K 


(4.45) 


1 


I 1 


* 


possessing  a positive  definite  solution  for  some  0 > 0. 


A 


3.  5 Stabilization  Through  Observers 


In  this  section  we  extend  the  previous  methods  to  the  decentralized 
stabilization  of  interconnected  systems  for  which  the  information 
available  to  the  subsystem  controllers  is  y^  = c..x^  rather  than  the  entire 
subsystem  state  x^.  In  this  case  the  subsystem  control  will  be  carried 
out  by  the  addition  of  subsystem  state  observers  [7],  We  consider  again 
the  subsystems 


x.  = A.x.  + B.u.  + v. 
a ii  ii  l 


y.  = C.x. 
yi  i i 


(4.46) 


for  i = I, . . . , N.  When  the  subsystems  in  (4.  46)  are  interconnected  we 
will  have  v^  = h.(xp.pc^j  t).  For  a suitable  control  scheme  on  each 
subsystem,  we  will  show  that  a condition  of  the  form 


N N 

y h.'h.  s y p . x.'x. 

r-\  ii  . , 'i  i i 


(4.47) 


will  again  guarantee  the  stability  of  the  interconnected  system.  To  do  this 
we  will  consider  the  dissipativeness  of  the  system  in  (4.46)  in  a suitable 
supply  rate  w.  For  simplicity  we  drop  the  subscript  i and  consider  the 


system 


Jc  = Ax  + Bu  + v 


(4.  48) 


where  (A,  B,  C)  is  controllable  and  observable.  If  v = 0,  (4.48)  can  be 


stabilized  by  first  building  an  observer  to  track  x by  a vector  x of  the 

# — 

same  dimension  as  x and  then  letting  u = Lx.  The  assumption  of 
controllability  and  observability  guarantee  that  the  poles  of  the  new  system 
can  be  placed  arbitrarily  [7]  and  thus  the  system  is  stabilizable.  In 
assessing  the  dissipativeness  of  the  system,  v is  not  null  and  some 
further  analysis  is  needed.  A diagram  for  an  observer -stabilizer  could 
be  as  follows 


The  design  parameters  here  are  the  matrices  P,  L. 
The  equations  for  this  system  are  then 


We  assume  a non-minimal  order  observer. 
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' L 


or 


_ / A + PC  0 \ _ / _I  \ 

x = Ax  + Bv,  x'  = (e',x'),  A = ( 1 , B = | 

' BL  A + BL  ' V I / 


Let  us  assume  that  an  observer  system  such  as  (4.49)  has  been  implemented 
for  each  subsystem  in  (4.46)  in  the  parameters  P. , L..  Furthermore,  each 
of  the  subsystems 


/ e.  \ /A.  + P.C.  0 \ / e.  x / -I  \ 

1 - ‘ ‘ ‘ )(')*{)*■ 
\ k.  ! \ B.L.  A.  + B.L.  / V x.  / \ I ’ 1 


(4.  50) 


i i 


1 IX 


is  assumed  dissipative  in  the  rate 


w(v.,x.)  = v.'v.  - p.  x.  (0  I)  (^)  x. 
' i’  i li  ri  i 'I'  i 


= v.'v.  - p . x.'x. 

li  ill 


The  behavior  of  the  interconnected  system  after  the  implementation  of  the 
observers  is  described  by  (4.  50)  for  i = 1,  . . . , N and  v.  = h.(x,  . . . , x^,*  t). 
In  view  of  our  previous  results,  the  entire  system  will  be  dissipative  in  the 


rate  w = £ w^  and  stable  provided 


N 


N 


• • • , t)  h.(xj,  • • • , Xj^j  t)  s p 

i=  1 i=  1 


This  is  a stability  condition  in  the  desired  form  (4.47). 

The  dissipative  ness  of  the  systems  in  (4.  50),  or  for  simplicity  that 
in  (4.49),  can  be  verified  by  using  the  methods  developed  so  far.  In 


particular,  (4.49)  will  be  dissipative  in  the  rate  w = v'v  - p x'x  provided 

(a)  A is  stable,  i.  e.  A + PC,  A + BL  are  stable 

(b)  The  (ARE) 

AK  + KA-p  (J)(I  0)  = K (-J)  (-1,  I)  K (4.  51) 

has  a negative  definite  solution. 

For  given  P,  L,  p both  (a)  and  (b)  can  be  checked;  the  B-algorithm 
will  be  used  for  (b).  The  (ARE)  (4.  51)  will  have  a solution  for  p = 0 and 
by  continuity,  for  all  p £ [0,  p q ].  The  upper  bound  p^  can  be  approximated 
to  a desired  accuracy  by  using  Algorithm  1. 

In  the  case  where  Ai  is  not  stable  for  some  i = 1,  . . . , N,  one  can 
choose  Pj , L.  such  that  (4.  50)  is  stable  [7].  This  will  guarantee  that 
(4.50)  will  be  dissipative  for  some  p>0.  Indeed,  the  (ARE)  (4.51)  will 
possess  a negative  definite  solution  for  p = 0 provided  A is  stable,  and 
by  continuity  this  property  will  persist  for  p g [0,  pQ]  for  some  pQ  >0  . 

We  can  always  obtain  a stability  bound  (4.47)  albeit  with  p.  small. 

The  question  of  designing  P.,  L.  in  an  optimal  fashion,  i.  e.  to 
guarantee  as  large  a as  possible  is  at  least  as  complicated  as  the 
analogous  problem  of  Section  3.  1.  The  design  of  P,  L should  be  done 
simultaneously  in  view  of  their  interdependence  as  evidenced  in  the  system 
matrix  A..  Such  a task  seems  prohibitive  at  the  moment.  A reasonable 
suboptimal  method  would  decouple  the  design  by  first  choosing  a L.,  on 
the  basis  of  the  game  theoretic  design  in  Sections  3.  2-4,  and  then  selecting 
an  observer  matrix  P such  that  A + PC  is  stable.  The  maximal  value 
of  p.  can  then  be  determined  by  an  application  of  Algorithm  1 for  the 
matrices  P.  and  L = - B'K+. 
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VALUES  AND  STRATEGIES  FOR  LINEAR  QUADRATIC  GAMES 


OF  INFINITE  DURATION 


Introduction  and  Preliminaries 


In  the  last  section  we  introduced  a zero  sum  differential  game  with 


value  functional 


and  dynamics 


Our  main  interest  was  in  finding  conditions  for  the  problem 


of  the  inf-sup  is  equivalent  to  the  stronger  statement 


i.  e.  , that  a finite  value  V(xn)  exists  for  the  game  (5.  1).  In  this  section 


we  verify  these  claims  and  examine  the  question  of  values  and  strategies 
for  the  infinite  duration  game  (5.  1),  a problem  interesting  in  its  own  right 
For  simplicity  we  consider  the  parameter  T]  in  (5.  1)  to  be  unity  without 
any  loss  of  generality  as  having  T)  / 1 is  equivalent  to  changing  the 


Perhaps  the  most  appealing  way  to  show  the  existence  of  the  value 


is  to  exhibit  an  equilibrium  pair  of  strategies.  This  is  a pair  u 


v such 
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J(u°,v)  ^ J(u°,v°)  <:  J(u,  v°) 


(5.4) 


for  all  u,  v6  U,  V the  set  of  admissible  strategies.  We  assume  through- 
out this  section  that  the  admissible  strategies  are  continuous  functions 
u(x,  t ) , v(x,t)  in  x,  t only.  It  follows  from  (5.4)  that  the  value  of  the 
T,  0 

game  is  J (u  , v ). 

For  a finite  duration  differential  game,  i.  e.  , one  with  value 


functional 


JT(u,v;x0)  = £ 


'C'Cx  + u'u  - v'v  dt 


(5.5) 


and  dynamics  as  in  (5.  2),  a pair  of  equilibrium  strategies  can  be  con- 
structed by  "completing  the  square.  " We  need  first 


LEMMA  5.1.  Consider  the  value  functional  (5.  5)  and  the  dynamics 


(5.2). 


(i)  Let  us  assume  that  the  Riccati  Equation  (RE) 


K + A'K  + KA+C’C  = K(BB'  - DD')K 


(5.6) 


with  the  end-point  boundary  condition  K(T)  = 0 has  a solution  on  [0,T] 
denoted  by  K(t,  T).  Then  for  any  two  controls  ufc,  v t € [0,T]  the 
value  functional  (5.  5)  subject  to  (5.  2)  becomes 


This  follows  from  the  inequalities 


J(u°,v°)  = sup  J(u°,v)  2 inf  sup  J(u,  v)  ^ sup  inf  J(u,  v)  ^ inf  J(u,v°) 


r/  0 0» 

= J(u  , v ) 
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JT(u,v;xQ)  = x^K(0,T)x0  + / ||  + B'K(t,  T)xt  ||  2 dt 

- J |j  vt  - D'K(t,  T)xt||2  dt  ( 

(ii)  Let  K be  a real  symmetric  solution  of  the  Algebraic  Riccati 
Game  (ARG)  equation 

A'K  + KA+C'C  = K(BB'  - DD')K  (! 

For  any  ut>  vt  * € [0,  T]  the  functional  (5.5)  becomes 


Jrp(Vlj  Vj  Xq)  — XqICXq  ~ XrplCXrp  t 


/ { ""I  + B’K-,I 


2 - ||vt  - D'Kxt||2  dt 


Expression  (5.9)  will  be  very  useful  for  the  infinite  duration  game. 
Expression  (5.  7)  gives  us  an  equilibrium  pair  of  strategies  for  game  (5.  5). 

PROPOSITION  5.1.  Under  the  assumptions  of  Lemma  5.  1 (i)  the 
strategies 


uT(xt)  = - B'K(t,  T)xt 
v°  (xt)  = D'K(t,  T)xt 


are  in  equilibrium  and  the  value  of  the  game  is  Vj,(Xq)  = x^K(0,  T)Xq. 


Lemma  1 and  Proposition  1 appear  in  [1]. 
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Proof.  Definition  (5.4)  of  equilibrium  pairs  is  immediately  verified: 
Using  (5.  7)  we  get 

J(u°,v;x0)  = x^K(0,T)x0-  £ j|  vfc  - D'K(t,  T)xt  ||  2 dt 

and  thus  v^,  is  the  optimal  strategy  for  v,  verifying  the  left  inequality  in 

(5.4).  The  right  inequality  is  proved  similarly.  Q.  E.  D. 

Consider  now  the  situation  where  lim^^  K(0,  T)  = K exists.  Then 

lim_  V„(xri)  = x' Kx_.  It  should  be  expected  that  the  value  of  the 
T-»°°  T l)  U U 

infinite  duration  is  V(xQ)  = x^KxQ  and  this  is  indeed  proven  in  Section  5.  3. 
Thus  the  existence  of  lim_,  K(0,  T)  will  guarantee  a finite  value  for 
game  (5.  1)  and  hence  a finite  inf^  supv  J(u,  vjx^).  Conversely,  if 
lim_  K(0,  T)  does  not  exist,  it  will  also  be  shown  in  Section  5.  3 that 

X“~*  ® ' 

inf  sup  J(u,  v:xn)  is  not  finite  for  all  x.  and  hence  the  finiteness  of  the 
u rv  7 1 0 U 

value  of  (5.  1)  is  equivalent  to  the  finiteness  of  the  inf-sup  J(u,  v,*Xq).  In 
the  next  Section  5.  2 we  develop  some  results  about  the  Riccati  type 
differential  equation  (5.6)  that  will  be  needed  in  5.  3. 

For  a more  game  theoretic  oriented  presentation  of  the  results 
in  this  Section  see  [5]. 

2.  The  Game -Riccati  Equations 

The  state  x enters  the  value  functional  (5.  1)  through  a positive 
semidefinite  matrix  C'C.  This  fact  guarantees  a simple  behavior  of  the 
solutions  of  the  (ARG)  (5.  6).  We  have 

LEMMA  5.2. Let  K(t,  T),  the  solution  of 
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K + A'K  + KA  + C'C  = K(BB'  - DD')K 


(5.6) 


with  endpoint  condition  K(T,  T)  = 0,  exist  for  t £ [0,T].  Then  K(t,  T) 
is  nonincreasing  in  t,  in  the  sense  of  positive  matrices. 

Proof.  We  give  a game  theoretic  argument.  In  view  of  the  time 
invariance  of  (5.  6)  it  suffices  to  show  that  K(0,  T)  ^ K(0j  T + a)  for  any 
A s 0 such  that  K(0,  T+A)  exists.  The  value  of  the  game  (5.5)  of 
duration  T+A  is  (Prop.  5.  1) 

VT+A(x0)  = xoK<°>T+A>xO  (5>11) 

Consider  a closed  loop  strategy  for  the  maximizer  in  this  game  defined 


v(xj  = 


I 


|D'K(t,T)xt 


t € [0,  T) 


t € [T,  T+A] 


Note  that  for  any  u,  x„  we  have  (Lemma  5.  1) 


ig  + B'K(s,T)xs||"  ds 


JT+A(U’^X0)  = X0K(°>  T)x0  + ^ II  us  + B'K(s,T)xs||2  ds 

T+a 

+ KC'Cxs  + UsUsj  ds  * x0K(0’T)x0 

However,  from  the  definition  of  value  and  (5.  11)  we  obtain 

x^K(0,T+A)x0  = VT+A(x0)  2s  inf  J(u,v;x  ) 

u 

a xqK(0,  T)xq 


A 


which  proves  the  lemma  as  x»  is  arbitrary. 


Q.  E.  D. 


Let  us  assume  now  that  K(0,  T)  exists  for  all  T >0  and  in  fact 
limT  w K(0,  T)  = K+.  In  view  of  the  time  invariance  of  (5.  6),  this  is 
equivalent  to  saying  that  K(t,  T)  exists  for  all  t ^ T and  K(t,  T)  = K 

Now  K(0,  T)  is  related  to  K(0,  T)  through  (5.  6)  and  thus  the  existence  of 
lim^,^  K(0,  T)  implies  that  lim^,^  K(0,  T)  = 0.  Therefore 
K+  = lim^_ ^ K(0,T)  satisfies  the  (ARG) 


A'K  + KA+C'C  = K(BB'  - DD')K 


(5.8) 


Furthermore  K(t,  T)  a 0 by  Lemma  5.  2 and  thus  K(0,  T)  s 0 and 
K+  ^ 0.  Thus  K+  is  a real  symmetric  positive  semidefinite  solution  of 
(5.8).  The  following  lemma  shows  that  K+  is  in  fact  positive  definite. 

LEMMA  5.  3:  Any  real  symmetric  solution  K of  (5.  8)  is  invertible, 
assuming  (A,  C)  is  observable. 


Proof.  Let  there  be  a x / 0 such  that  Kx  = K'x  = 0.  Post-  and 
pre multiplying  (5.8)  by  x and  x'  we  get  that  x'C'Cx  = 0 and  thus 
Cx  - 0.  By  just  postmultiplying  (5.8)  by  x we  get 

A'Kx  + KAx  + C'Cx  = K(BB'  - DD')Kx 

and  hence  KAx  =0.  If  Ax  = 0,  we  have  Ax  = 0,  Cx  = 0 contradicting 
the  observability  assumption.  If  Ax  / 0,  by  repeating  the  above  arguments 
with  Ax  in  place  of  x we  obtain  CAx  = 0,  CA^x,  . . . , CAnx  = 0, , 
which  again  contradicts  the  observability  assumption  unless  x = 0.  Q.E.  D. 


t 


We  showed  therefore  that  if  lim, 


positive  definite  solution  to  (5.  8).  T-he  converse  is  also  true.  To  prove 
this  we  first  provide  a bound  for  K(t,  T). 


LEMMA  5.4 


K = K'  > 0 be  a real  solution  to  (5.  8).  Then 


Proof.  We  give  a game -theoretic  proof.  Let  us  assume  that  the 


B'Kx.  The  value  functional  for 


(5.  5)  of  duration  T,  ^ T can  be  written  for  any  v as 


It  follows  from  the 


assumption  K > 0 that  for  all  v,  J 


The  value  of  game  (5.  5)  is  thus  less  than  x'Kx.  and  we  conclude  from 


PROPOSITION  5.  2:  Under  the  assumption  of  observability, 

i0D  K(0,  T)  exists  iff  there  exists  a positive  definite  solution  to  (5.  8) 
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Proof.  If  K = K'  > 0 is  a solution  to  (5.  8),  then  K(t,  T)  exists  for 

any  T and  any  t € [0,  T].  This  follows  from  the  fact  thaf  K(t,  T)  is 

nondecreasing  as  t decreases  and  can  fail  to  exist  only  if  it  increases 

without  a bound.  This  cannot  happen  in  view  of  Lemma  5.4.  Thus  K(0,  T) 

exists  for  all  T,  and  in  fact  it  is  increasing  in  T and  bounded  by  K. 

Thus  lim_,  K(0,  T)  exists.  The  converse  has  been  proven  already. 

00  7 

Q.  E.  D. 

Proposition  5.  2 implies  that  K+  = lim^^  K(0,  T)  is  the  smallest 
positive  definite  solution  of  (5.  8).  Indeed,  if  K = K'  > 0 is  a solution  of 
(5.  8),  K ^ K(0,T)  and  hence  K a lim^,^  K(0,  T)  = K+.  This  observation 
is  used  to  give  an  important  property  of  K+. 

LEMMA  5.  5:  K+  is  feedback  stabilizing,  i.  e.  Re  X.(A-BB'K+ 

+ DD'K+)  £ 0. 

Proof.  Note  that  (K+)  ^ exists  and  satisfies  the  (ARG) 

-AM  - MA'  + (BB1  - DD')  = MC'CM  (5.  14) 

According  to  Theorem  A.  1.  1 of  Appendix  1,  there  exists 
a maximal  solution  M+  of  (5.  14),  i.  e.  , such  that  M ^ (K+)  > 0. 

Furthermore  M*  satisfies  Re  X(-A'  - C'CM^)  £ 0.  Now  0 < (M^)  £K 
and  (M+)-1  is  a solution  of  (5.8),  contradicting  the  minimality  of  K+ 
among  the  positive  definite  solutions  of  (5.  8)  unless  (M+)  ^ = K+.  By 
rearranging  (5.  14)  we  obtain  that 

( -A'  - C’CM+)  = K+(A  - BB'K+  + DD'K+)(K+)-1 


•m&saiMsm?, 
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and  hence 


Re  \(A  - BB'K+  + DD'K+)  = Re  \(-A'  - C'CM+)  < 0 


Q.  E.  D. 


3.  The  Value  of  the  Infinite  Duration  Game 

In  this  section  we  assume  that  there  exists  a solution  K = K'  > 0 
to  (5.  8).  It  follows  from  Proposition  5.  2 that  lim^^  K(0,  T)  = K+.  In 
view  of  the  time  invariance  of  (5.  6)  we  also  have  lim^  K(t,  T)  = K+ 
for  any  fixed  t.  In  order  to  show  that  v(xg)  = Xg^+XQ  is  t*ie  value  of 
game  (5.  1)  it  would  suffice  to  exhibit  an  equilibrium  pair  of  strategies, 
as  was  done  in  Proposition  5.  1 for  finite  duration  games.  In  fact  the 
limit  of  the  equilibrium  strategies  for  the  finite  duration  game  uJ^Xx^), 
v^,(xt)  as  T -*  <*>  become 


0 + 
u (xt)  = - B'K+xt 

v°(xt)  = D’K+xt 


(5.  15) 


. i 


and  are  obvious  candidates  for  being  in  equilibrium.  It  is  mildly 
...  00 

surprising  that  u , v need  not  be  in  equilibrium.  In  fact,  consider  a 
scalar  game  with  value  functional 


J(u,  v;xQ)  = J x2  + u2  - 


v2  dt 


and  dynamics 


X = X + u + -=  V 

iT2 


x(0)  = x. 


1 i 
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The  Riccati  differential  equation  is 


• 1 2 
k = i k - 2k  - 1 


k(T)  = 0 


which  can  be  integrated  to  give 


k(t,  T)  = -JU  tanh  [^(T-t)  + tanh  *(-2/*y6)]  + 2 


Now  lim^^  k(t,  T)  = 2 + Jb  and  the  strategy  pair  in  (5.  15)  is 
u^*(xt)  = - (2  + N/^)xt  anc^  v^(x^)  = (2  + \ JE/  JZ)x^.  These  strategies  are 
not  in  equilibrium.  If  the  maximizer  uses  the  strategy  v°(xt),  the 
minimizer  is  faced  with  the  least  squares  problem 


Inf 
u "0 


J * x2  + ^2  /„0/..\\2  ^ _ | r,  A 0 2 , 2 


(vu(x))^  dt  = J [(-4  - 2 n/?)x^  + u^]  dt 
• 0 


and  dynamics 


x = 


x + u + ^ V°(x) 


4 + J6 

~2  ' x + u 


x(0)  = x. 


The  minimizer's  optimal  response  is  not  u^(xt>.  In  fact,  by  playing 

0, 


u(x^)  = 0 the  value  functional  becomes  J(0,  v (x);xQ)  = - ® for  x^  / 0. 
Thus  the  right  hand  inequality  in  the  definition  of  equilibrium  strategies 
(5.4)  is  not  satisfied. 

We  next  prove  that  x^K+Xq  is  nevertheless  the  value  of  game  (5.  1) 
by  invoking  the  definition  of  value  (5.  3)  directly  instead  of  exhibiting  an 
equilibrium  pair.  Such  a pair  need  not  even  exist  for  this  game. 


•• 


1 
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PROPOSITION  5.3:  Let  (A,C)  be  observable  and  let  K = K'  > 0 

be  a solution  of  (5.  8).  The  value  of  game  (5.  1)  is  V(xQ)  = X()K+X0’ 

Proof.  We  have  to  show  according  to  (5.  3)  that 


inf  sup  J(u,  v,-xQ)  = sup  inf  J(u,  v;xQ)  = x^K+xQ 


u v 


v u 


where  the  inf,  sup  are  taken  over  the  closed  loop  strategies.  This  is 
equivalent  to  showing  that  for  any  e>0  there  exist  strategies  u.^ (x) , 


vg(x)  such  that 


J<ue*v;xo)  * X0K  x0  + 6 


V v 


(5.  16) 


JK  ve’xo)  2 xoK+xo  ' € 


V u 


(5.  17) 


We  know  that  lim„  K(0  T)  = K+  exists.  Consider  the  strategy 
£-♦  00 

u^(x^)  = - B'K+x^  We  will  show  that  it  satisfies  (5.  16)  for  any  e.  In  fact, 
consider 


sup  J(u  ,v;x  ) =-inf 


J 


x'(C'C  + K+BB'K+)x  + v'v  J dt 


v 0 


x = (A-BB'K  )x  + Dv  x(0)  = x 


0 


\ 


The  (ARE)  for  this  problem  is 


(A-BB'K+)'M  + M(A-BB'KT)  - C'C  - KTBB'KT  = MDD'M  (5.  18) 


Note  that  (5.  18)  holds  for  M = -K+  <•  0 and  in  addition  -K+  is  feedback 
stabilizing  for  this  problem,  i.  e. 
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Re  \(A-BB'K+  - (-DD'K+))  = Re  X(A-BB'K+  + DD'K+)  s 0 


according  to  Lemma  5.  5.  It  follows  from  Theorem  A.  1.  3 that  the 


infimum  is  - x^K+xQ  or  supv  J(u°,  v;  xQ)  = x^K+xQ  which  proves 


(5.  16)  with  e = 0. 

To  construct  a vg(xt)  satisfying  (5.  17)  for  a given  e and  Xq  we 
work  as  follows.  There  exists  a T = T(g,Xq)  such  that 


xiK+xo  " e s xoK(0;T)xo 


(5.  19) 


This  follows  from  lim^,^  K(0,  T)  = K . Consider  now  the  strategy 


D'K(t;  T)x 


ve(t)  = 


te  [o,  t) 
[T,») 


(5.  20) 


This  corresponds  to  playing  optimally  for  a game  of  duration  T for 
t € [0,  T)  and  using  no  control  from  then  on.  We  denote  the  integrand  of 
the  functional  (5.  1)  as  w(u,  v,  x)  and  split  the  integral  as 


J( 


u’ve’x0*  = j WK  ve>x)  dt  = ^ w(u,ve,x)  dt  + f w(u,  v€, 


x)  dt 
(5.21) 


The  last  integral  in  (5.  21)  is  nonnegative  by  the  definition  of  v£  (x),  i.  e.  , 


it  equals  JZ  [x'C'Cx  + u'u]  dt. 


We  can  express  the  first  integral  according  to  Lemma  5.  l(i)  as 


% 


r 
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J w(u,ve,x)  dt  = x^K(0,  T)xQ  + ^ |jut  + B'K(t,  T)xt| 


dt 


1 

/ 


jve  - D'K(t,T)xt|r  dt 


(5.22) 


The  last  integral  vanishes  by  the  definition  of  v (t)  and  thus 

v 


T 

/ 


w(u,  vc>x)  dt  a xoK(°,T)x0 


Hence  J(u,  vc;xQ)  2 x^K'O,T)x0.  Using  (5.  19)  we  finally  obtain 


J(u,  vc;x0)  * XqK(0,  T)xq  2 XqK+xq  - € 


which  is  exactly  (5.  17).  Q.  E.  D. 

\ converse  to  Proposition  5.  3 would  state  that  game  (5.  1)  does  not 
have  a value  if  the  (ARG)  (5.  8)  does  not  have  a positive  definite  solution. 

The  nonexistence  of  a positive  definite  solution  of  (5.8)  implies  that 
lim^  ooK(0,  T)  does  not  exist.  Since  K(0,  T)  is  nondecreasing  in  T it 
follows  that  XqK(0,  T)Xq  increases  without  bound  in  T (for  some  x^  / 0), 
i.  e.  , for  any  M > 0,  there  exist  a T such  that  XqK(0,  T)xq  >M.  We  can 
use  this  fact  to  show  that  inf^  suPy  J(u.,  v;  ) 2 sup^  inf^  J(u,  vjx^)  > M. 
For  a strategy  vj^(x)  defined  by  analogy  to  v^x)  in  (5.  20),  i.  e. 


vM(xt)  = 


D'K(t,  T)xt  t € [0,  T) 

0 t € [T,~) 


t 


f 


we  showed  that 


L-- 
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J(U’VM;X0)  2 x^K(0,T)x0>  M 

Hence 

sup  inf  J(u,  v;  xQ)  £ inf  J(u,  vM;xQ)  > M 
V U u 

and  since  M is  arbitrary  it  follows  that  both  infu  sup^  and  sup^  inf^ 
are  infinite.  This  shows  that  the  game  does  not  have  a finite  value  and 
also  shows  that  the  finiteness  of  the  inf-sup  problem  is  equivalent  fo  the 
finiteness  of  the  value,  as  was  claimed  in  Section  4. 

Remark  1.  The  above  results  were  derived  on  the  assumption  of 
prior  commitment  of  the  players  to  their  respective  closed  loop  strategies. 
The  players'  role  is  simply  to  implement  mechanically  a memoryless 
strategy  whose  only  input  is  the  observation  of  x.  After  the  implementation, 
the  players  let  the  game  proceed  without  further  interference.  The  situation 
would  be  much  different  if  the  players  were  allowed  to  change  strategies 
during  the  game,  say  at  time  t,  on  the  basis  of  the  entire  history 
{x(t),  0 s:  t ^ t}. 

The  examination  ot  games  where  prior  commitment  does  not 
hold  is  equivalent  to  expanding  the  strategy  sets  of  the  players  by  allowing 
the  control  at  time  t to  depend  on  the  available  memory,  i.  e.  part  of  the 
past  observations  (x(t),  T€[0,t]j  the  controls  applied  {v(t),t€  [0 , t ] } 
etc.  Even  in  this  expanded  formulation  the  existence  of  K+  implies  that 
the  value  of  the  game  is  V(xQ)  = x^K+xQ,  since  each  player  can  get 
e-close  to  that  value  regardless  of  the  control  time-function  employed  by 
his  opponent.  An  interesting  problem  is  to  find  whether  equilibrium 
strategies  exist  in  the  expanded  strategy  space  where  some  kind  of  memory 


is  allowed. 


Remark  2.  We  examine  in  some  more  detail  why  the  strategies 
u (x)  = -B'K+x,  v^(x)  = D'K+x  fail  to  be  in  equilibrium.  The  first  half 

of  the  proof  of  Proposition  5.  3 shows  that  u^(x)  is  a desirable  strategy 
since  sup^  J(u  , VJ  xq)  = xqK+Xq,  i.  e.  u^(x)  guarantees  that  the 
minimizer  attains  the  actual  value  of  the  game.  In  fact,  if 
Re  X.(A-BB'K  + DD'K^)  < 0 the  proof  in  Proposition  5.  3 can  be 
strengthened  to  show  that  v^(x)  is  the  optimal  response  to  u^(x).  How 
ever,  the  situation  is  not  always  good  for  the  maximizer  if  he  commits 
himself  to  v (x).  The  scalar  example  examined  earlier  shows  that  the 
minimizer  can  take  advantage  of  this  commitment  to  drive  J to  -®  . 
Consider  the  general  problem  faced  by  the  minimizer  when  he  knows  that 
the  maximizer  is  committed  to  v°(x)  = D'K+x 


00 

inf  J x'[C'C  - K+DD'K+]x  + u'u 


u 0 


x = (A  + DD'K+)x  + Bu 


x(0)  = xr 


(5.23) 


The  corresponding  (ARG)  is 


(A  + DD'K+)'M  + M(A  + DD'K+)  + CC1  - K+DD'K+  = MBB'I 


(5.  24) 


and  (5.  23)  will  have  a finite  infimum  iff  (5.  24)  has  a nonpositive  definite 
solution.  It  is  easy  to  check  that  M = K+  is  a solution  of  (5.  24)  and  is 
in  fact  the  maximal  solution  since  K is  feedback  stabilizing.  Further- 
more if  (5.  23)  possesses  a negative  definite  solution,  the  infimum  will  be 
x0^  xq  anc^  will  be  attained  by  u = - B'K  x.  These  remarks  rely  on 
Theorem  A.  1 of  Appendix  1.  Thus  a necessary  and  sufficient  condition  for 


J»  * • V ft* 


u°(x)  = -B'K+xt  and  v°(x)  = D'K+x  to  be  in  equilibrium  is  that 


(i)  Re  \(A  - BB'K+  + DD'K+)  < 0 


(ii)  There  exists  a negative  definite  solution  to  (5.  24) 


(5.  25) 


Condition  (5.  25)  is  perhaps  unsatisfactory  since  it  involves  K in  addition 
to  the  parameters  of  the  game.  In  view  of  the  difficulty  of  finding  alter- 
native conditions  for  a least  squares  problem  to  be  finite  (See  [4]),  it 
seems  that  (5.  25)  are  as  good  conditions  as  one  can  hope  for. 


Remark  3.  A reason  for  the  failure  of  v (x)  is  that  it  might  pay 
for  the  minimizer  to  drive  the  state  x far  from  the  origin  if,  in  addition, 
he  is  ensured  of  a negative  value  functional.  The  strategies  v£(x)  guard 
against  this  by  stipulating  that  v = 0 for  large  t.  The  minimizer  faces 
the  positive  functional  J“{x'C'Cx  + u'u]  dt  and  he  will  try  to  drive  x 
back  to  the  origin.  In  view  of  this  remark  one  could  conjecture  that  a 
simple  set  of  e -optimal  strategies  for  v could  be  obtained  by  truncating 
v^(x)  = D'K+X  for  large  T,  i.  e.  using 


vT(xt)  = 


| D'K+xt 


t e [0,  T) 


t € [T,  “) 


(5.26) 


The  conjecture  is  that  for  large  T,  J(u,  v^JXq)  £ x^K  xq  “ 6 f°r  u- 


Consider 


1 
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inf  J(u,  vT;xQ) 
u 


*P  00 

w(u,  v°(x),x)  dt  + 


x)  dt  + J x'C'Cx  + u'u  dt 


x = (A  + DD'K+)x  + Bu  x(0)  = xQ;  for  t£[0,T)  / (5.27) 


x = Ax  + Bu  for  t € [T,00) 


. 0, 


For  time  T and  on,  the  optimal  response  of  the  minimizer  is  u (x)  = -B'Px 
where  P = P'  > 0 is  the  solution  of 

A'P+PA+C'C  = PBB'P 


and 


inf  f { x'C'Cx  + u^(x)'  • u^(x)}  dt  = xLPxT 
u T 


Using  the  expressions  in  Lemma  5.  1 we  can  write  the  functional  in  (5.  27)  as 

T_ 

II..  x II  ^ ji-  x ^ ro  _ 

T 


Tr 

• J(u,  vT;xQ)  = x^K+x0  + / II  ut  + B'K+xt||2  dt  + xT(P  - K+)X(] 


and  making  the  transformation  u^  = u + B'K+x  we  finally  reduce  (5.  27)  to 
T 


dt  + xT(P  - K+)xt  + XqK+Xq 


inf  J u'u 
u "O 

k = A+x  + Bu  = (A  - BB'K+  + DD'K+)x  + Bu  x(0)  = xr 


(5.  28) 


The  value  of  the  infimum  is  XqK(0,  T)Xq  provided  a solution  K(t,  T)  of 


K + (A+)'K  + KA+ 


= KBB'K 


(5.29) 
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with  the  endpoint  condition  K(T,  T)  = P - K+  exists  on  [0,  T). 

The  stabilizing  solution  of  the  (ARG) 

(A+)'K  + KA+  = KBB'K  (5.30) 

corresponding  to  (5.29)  is  K | = 0 provided  Re  MA+)  £0.  According  to 
Remark  21,  [3]  lim,p_>a;  K(0;T)  = k|  - 0 provided 


P - K+  > Kj 


(5.31) 


where  Kj  is  the  minimal  solution  of  (5.  30).  Thus  if  (5.  31)  holds, 
lim  j._x  J(u,  v^,;xQ)i  XqK  Xq  regardless  of  u.  The  strategies  v^(x) 
of  (5.  26)  are  more  appealing  than  the  v .(x),  being  a truncation  of  time- 
invariant  linear  strategies  whereas  V(  (x)  is  time-varying  in  an  intricate 
fashion.  It  should  be  noted  though  that  (5.  31)  does  not  always  hold  and 
hence  the  v^,(x)  strategies  are  not  always  applicable,  as  for  instance  in 
the  following  example: 

T/  . f 2 2 2 

J(u,  V>  xr>)  “ J X -)-  U - V dt 
0 

X - X + U + yj  V x(0)  r Xq 

The  (ARG)  for  this  problem  is  2k  + 1 = 1/3  k^ 


k+  = 3 t Jl2,  A+  = - 
In  addition,  p > 0 is  the  solution  of 


yfH  u-  _ 2^ 

3 ’ 1 " “ 3 

2p  + 1 = pZ  i.  e.  , p = 1 + Vzl 
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f 


I) 


Condition  (5.  31)  is  violated  since 


(1  +yj2)  - (3  -^12)  i.  - \ Vl2 


By  using  similar  calculations  as  above  it  can  be  shown  that 


0 + 

sup  J(u,  v (x);x  ) = x'K  x 

u u 


provided 


- kx  > k; 


(5.33) 


The  fact  that  P > 0 shows  that  (5.  31)  might  hold  even  if  (5.  33)  does  not. 


The  usefulness  of  the  v^(x)  strategies  is  evident  since  they  can  be 


0, 


successful  even  when  v (x)  is  not 


\ 


-135- 


REFERENCES 


1.  Banker,  M.  D.  , "Observability  and  Controllability  of  Two  Player 
Discrete  Systems  and  Quadratic  Control  and  Game  Problems,  " 

Ph.  D.  Thesis,  Stanford  University,  May  1971. 

2.  Brockett,  R.  W.  , Finite  Dimensional  Linear  Systems,  Wiley,  New 
York,  1970. 

3.  Willems,  J.  C.  , "Least  Squares  Optimal  Control  and  the  Algebraic 
Riccati  Equation,  " IEEE  -AC,  Vol.  16,  No.  6,  Dec.  1971. 

4.  Willems,  J.  C.  , "On  the  Existence  of  a Nonpositive  Solution  to  the 
Riccati  Equation,  " IEEE -AC,  Vol.  19,  No.  5,  Oct.  1974. 

5.  Mageirou,  E.  , "Values  and  Strategies  for  Infinite  Duration  Linear 
Quadratic  Games,  " to  appear  in  IEEE  -AC,  Vol.  21,  August  1976. 


i 


APPENDIX  1 


LEAST  SQUARES  PROBLEMS  AND  THE 
ALGEBRAIC  RICCATI  EQUATION 


We  summarize  here  some  results  in  [1],  [2]  on  the  relation  between 
the  least  squares  problem 


V^(Xq)  = inf  J {x'Qx  + 2u'Px  + u'u]  dt 
u i) 


x = Ax  + Bu  x(0)  = x_  ; (A,  B)  controllable 


the  algebraic  Riccati  equation  (ARE) 


A'K  + KA  + Q = (KB  + P')(B'K  + P) 


(A.  1) 


(A.  2) 


and  the  frequency  domain  inequality  (FDI)  for  cu  £ R 


H( -jcu  , ju>)  = I + P(jc  I - A) ^B  + B'(-jtul  - A')_1P' 
+ B'(-jujI  - A')Q(ju)I  - A)_1B  £ 0 


(A.  3) 


THEOREM  A.  1:  (1)  The  (ARE)  (A.  2)  has  a real  symmetric 
solution  K = K'  if  and  only  if  the  (FDI)  (A.  3)  holds.  Under  this 
condition  there  will  exist  a unique  maximal  solution  K+  and  a unique 
minimal  solution  K of  (A.  2)  such  that  for  any  solution  K = K' 


K~  £ K £ K 

In  addition,  if  A*  - A - B(B'K+  + P),  A~  - A - B(B'K"  + P)  then 
Re  \(A+)  * 0,  Re  \(A_)  s 0. 

(2)  The  following  relations  are  equivalent: 
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(i)  H(-j<u,  j»)  2 e B'(-ju>I  - A')  ^(jujl  - A)_1B;  e>0 

(ii)  Re  \(A+)  < 0 

(iii)  Re  X.(A  ) > 0 

(iv)  K+  - K"  > 0 . 

(3)  Consider  the  infimum  Vf(xQ)  in  (A.  1).  It  is  finite,  Vf(xQ)  > -«, 

if  and  only  if  K~  s 0.  If  K"<0  V£(x0)  = x'qK+ Xq.  If  K- < 0,  K'<K+ 

the  infimum  is  attained  by  the  stabilizing  control  u = -(B'K  + P)x. 

(4) .  Consider  the  least  squares  problem  in  (A.  1)  with  the  additional 
constraint  lim£^^  x£  = 0.  Denote  the  value  of  the  infimum  by  V(Xq). 
Then  V(Xq)  > - • if  and  only  if  the  'ARE)  (A.  2)  has  a real  symmetric 
solution,  in  which  case  V(xQ)  = x^K+xQ.  If  in  addition  Re  \(A+)  < 0 

(see  2)  the  infimum  is  attained  by  the  stabilizing  control  u = - (B'K  + P)x. 

These  results  are  proven  in  [2].  For  the  case  where  P = 0, 

Q = -C'C  we  have 

THEOREM  A.  2 [1]:  Let  (A,  B,  C)  be  controllable  and  observable 
and  that  Re  \(A)  < 0.  Then  the  (ARE) 


A'K+KA-C'C  = KBB'K 


(A.  4) 


has  a unique  negative  definite  solution  K+  such  that  Re  X.(A+)  s 0 if  and 


only  if  the  following  (FDI)  holds  for  all  <u  g R 

I - B'(-j»I-A,)-1C,C(j«I-A)"1B  2 0 


(A.  5) 


i ' 


Theorem  A.  2 plays  an  important  role  in  the  proof  of  the  more 
general  results  of  Theorem  A.  1.  It  is  particularly  important  in  the 
development  of  Section  IV. 
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However,  the  positive  definiteness  conditions,  and  hence  K=  K'>  0, 
are  satisfied  for 


0 <:  p < 1/2 


(A.  6) 


We  present  some  computer  results  on  the  application  of  Kleinman's 
algorithm  to  (A.  2).  The  printout  includes 

(i)  P = ri'1 

(ii)  The  "Cost  Matrix'  BB'  - pi 

(iii)  The  "Solution  Matrix".  (M  ) 

’ l 

(iv)  The  "Error  Matrix",  i.  e.  the  1.  h.  s.  of  (A.  1)  with  K = (M.)"1. 

According  to  Proposition  3.  58  the  M. 's  are  decreasing  in  i.  Hence  our 
"solution  matrices"  are  increasing  in  every  iteration,  being  equal  to 
(Mjf1. 


The  exact  solutions  K+(p),  0 ^ ps  . 5,  as  given  by  (A.  3),  (A.  4)  are: 

K+,0)  ' (U ) 


Kr(.  45)  = 


K+(.  498)  = 


/ vT20 

10 

\ 10 

>7110 

/ 251 

250  \ 

\ 250 

251/ 

10.  955 
10 


10 

10.  955 


For  0.  5sps  .75  the  iterates  Mj  are  not  positive  definite.  The 
algorithm  still  converges.  The  exact  solution  for  p = . 7 is  , 


R 

1 
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Finally,  when  p is  set  above  . 75  the  numerical  procedure  becomes 
unstable.  In  our  implementation  of  Kleinman's  algorithm,  the  Lyapunov 
equat:',  an 


A'K  + KA  = - Q 

is  solved  by  Davison's  method  (*)  which  essentially  computes 


K 


00 


exp(A't)Q  exp(At)  dt 


If  A is  not  a stable  matrix  this  computation  will  be  infeasible.  In  view 
of  Proposition  3.58,  the  instability  of  A - BB'K.  for  some  i shows 
that  the  corresponding  Riccati  equation  does  not  have  a strictly 
stabilizing  solution. 


\ 


i 


(*)  Davison,  E.  and  Man,  F.  , IEEE-AC,  Vol.  AC-13,  p.  448. 


DATA 


A MATRIX 


CONTROL  MATRI 


BB* -MATRIX 


INITIAL  CONTROL 


COST  MATRIX 


RESULTS 


ITERATION  NO.  1 


SOLUTION  MATRIX 


9273314881246 
231881735 7 8C3 


2318817357862 
8 3 7 9 6 0 0 C 9 9 7 4 4 


A P.D.  MATRIX 


ERROR  MATRIX 


609582128518 

498049499134 


498049499134 

66017222G3U3 


*>ii^ 


ITERATION  NO.  2 


SOLUTION  MATRIX 

1.49 70 065 19 35 A .7372599491449 
. 737259949145  1.418589460237 

A P.D.  MATRIX 

ERROR  MATRIX 

9. 05529655 E- 02  -8 . 64535821E-02 
8. 64535821E-02  -.  115317895755 

ITERATION  NO.  3 
SOLUTION  MATRIX 

1.722541147259  .9875403793642 

.987546379363  1.714852474011 

A P.D.  MATRIX 

ERROR  MATRIX 

2 .02742492E-03  - 3 . 50  7 18  69 9E-0 3 
3 . 50 7 186  9 9 L-0  3 -6 . 97710066E-03 


ITERATION  NO.  4 


SOLUTION  MATRIX 

1.732031276368  .9999741012953 

.999974161295  1.732003691797 

A P.D.  MATRIX 


LRROR  MATRIX 

3.99  519000L-06  - 1 . 60496010E-00 
1 . 6 649 59 90L-0  6 - 3 . 39258210E-05 


COST  MATRIX 


RESULTS 


ITERATION  NO.  1 


SOLUTION  MATRIX 


2.133322277379 

1.066655911961 


1.0 06655911962 
1.823641342767 


A P.I).  MATRIX 


LRROR  MATRIX 


2640644173277 

2179704711008 


217970471100 

200787436299 


ITERATION  NO.  2 


OLUTION  MATRI 


A F.l).  MATRIX 


LRROR  MATRIX 


2 . 92906686E-02 
4 . 32714 9 41 E- 02 


ITERATION  NO.  3 


SOLUTION 


D.  MATRIX 


-mu 


ERROR  MATRIX 


r 
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■1. 9944326c E- 04  -5. 44957521 E- 04 
•5. 44957521 E- 04  -2 .09994619L-03 


ITERATION  NO.  4 
SOLUTION  MATRIX 


10.95430638835 

9.999804967349 


9. 9 9980491)  7345 
10.9541918984 


A P.D.  MATRIX 


ERROR  MATRIX 

1.822  737  00L-06  8 . 35C88800E-07 

8 .35689000E-07  -1 . 11999130E-05 


ITERATION  NO.  5 
SOLUTION  MATRIX 

10.95443125571  9 .999940x72555 

9.999940172543  10.95434174828 


A P.D.  MATRIX 
ERROR  MATRIX 

1. 90171900 E- 06  5 .05507600E-07 

5 .05306800L-07  -8 . 00926500E-06 


1 

1 


1 


i 
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P=  .498 

COST  MATRIX 

-.498  0 

0 .502 

RESULTS 

ITERATION  NO.  1 
SOLUTION  MATRIX 

2.279190460644  1.13953964021 

I .  139589  646211  1.90134316303 

A P.D.  MATRIX 
ERROR  MATRIX 

-.2805041512517  -.2337505155757 
-.2337305155754  -.20300703908 

ITERATION  NO.  2 
SOLUTION  MATRIX 

II. 29227204975  9 .864310617694 

9.864313617693  10.40485104973 

A P.D.  MATRIX 


> 


ERROR  MATRIX 


-2.51389314E-02  -3 . 424G0840E-02 
-3.42460840E-02  -4 . 95421700E-02 


U 

* 

} 
i I 

* I 

! 

9 . 

1 i 


ITERATION  NO.  3 
SOLUTION  MATRIX 

148.4825933774  147.1637329074 

147.1637329675  147.8376117229 


A P.D.  MATRIX 
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ERROR  MATRIX 

-2. 97769232 E- 04  -7 . 6860 J843E-04 
-7. 88505842 E- 04  -3 . 11780574E-03 

ITERATION  NO.  4 
SOLUTION  MATRIX 

250.770740138  249.7708493307 

249.7708493361  250.7669012672 

A P.D.  MATRIX 


ERROR  MATRIX 

1 .95295300L-06  1 . 1373G000E-06 

1 .13729880E-06  - 1 . 5482 2 30GE- 05 


NO.  5 


ITERATION 

SOLUTION  MATRIX 

250.9970560805 
249.9978118344 

A P.D.  MATRIX 

ERROR  MATRIX 

2 .46498700E-06 
6.19711000E-08 

ITERATION 


249.9978113347 

250.9945733825 


6 . 19  7 17000E-08 
-7 . 5 39 47000 L- 06 


NO.  6 


SOLUTION  MATRIX 

250.9968528565  249.9976094185 

249.9976094189  250.9943717819 

A P.D.  MATRIX 

ERROR  MATRIX 

2. 46175000 E-06  6 . 09831000E-08 

0. 09839000 E- 08  -7 . 58425300E-06 


COST  MATRIX 


RESULTS 


ITERATION  NO.  1 
SOLUTION  MATRIX 

3.199933347867  1.599983833841 

1.599983833841  2.338441689366 

A P.D.  MATRIX 

ERROR  MATRIX 

-.3812507316127  -.2843768342542 
-.284376884254  -.228129351652 


ITERATION  NO.  2 
SOLUTION  MATRIX 

-3.512988727674  -4.233052533967 
-4.233052583961  -2.535131697757 


NOT  A P.D. MATRIX 
ERROR  MATRIX 

-5. 700970 19 E- 02  -6. 69855371L-02 
-6. 69855371 E- 02  -8 . 74694405E-02 


ITERATION  NO.  3 

SOLUTION  MATRIX 

-1.32b607664785  -2.6143039422 
-2.614303942198  -1.249297734187 


NOT  A P.D. MATRIX 
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ERKOK  MATRIX 

-3.29340872E-03  -4 . 998 859 2 3E-0 3 t 

-4 .99885925E-03  -2 . 0o4l5 342E-0 2 

I 

ITERATION  IJO.  4 
SOLUTION  MATRIX 

-1.126436499108  -2 .50:>lo642i,247 
-2.505166426244  -1.126800150841 

LOT  A P.D. MATRIX 
ERROR  MATRIX 

-2.798926b3L-04  2 . 1 358444  6E-04 

2 .13584445E-04  -1 . 0029 0206L-0 3 

ITERATION  UO.  5 

SOLUTION  MATRIX  \ 

-1.118086249761  -2 . 500012449186 
-2.500012449136  -1.118040270598 

t 

NOT  A P.D. MATRIX 
ERROR  MATRIX 

2. 18454000 E- 06  - 3 . 34096080E-06 
-3. 34096070 E-Ob  -7. 54920700E-06 

ITERATION  NO.  6 
SOLUTION  MATRIX 

-1.118056363739  -2.499987532182 
-2.499987532183  -1.118000649691 

NOT  A P.D. MATRIX 
ERROR  MATRIX 

4. 49 4 79000 E- 06  -4 . 44695460E-06 
-4 .44695460L-06  -5 . 51742100E-06  f 


....  _ 


.. 


gjiidMWJWBwwwar1 


W 


— 
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P-  .751 

COST  MATRIX 

-.751  0 

0 .249 

RESULTS 

ITERATION  NO.  1 
SOLUTION  MATRIX 

-1.629107234723  -2 . 7 37883 7c8 6 12 
-2.737883768609  -1.218498805j.36 

NOT  A P.D. MATRIX 
ERROR  MATRIX 

-5. 30 89 116 IE- 02  -3. 89029555E-02 
-3. 89029555 E- 02  -8 . 52059851E-02 

ITERATION  NO.  2 
SOLUTION  MATRIX 

-. 556780572646  -2 . 10S40422913 

-2.10840422913  -. 51876504469 2 


NOT  A P.D. MATRIX 
ERROR  MATRIX 

-9. 3 7 79 1271 E-0 3 -2 . C9752505E-03 
-2. 69752505 E- 03  -2 .62481507E-02 


ITERATION  NO.  3 
SOLUTION  MATRIX 

l -.24413349532  -2.026846204454 

-2 .020846204455  -.2503422792094 


• NOT  A P.D. MATRIX 

ERROR  MATRIX 

-3. 9 9 89 6 35 2 E-0 3 8 . 68285063E-04 

8 . 68285065E-04  -5 .468005 j3E-03 
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ITERATION  NO.  4 
SOLUTION  MATRIX 

-8.94044002L-02  -2 . 00420t>4l9456 
-2.004206419437  -8 . 98844035E-02 

NOT  A P.D. MATRIX 
ERROR  MATRIX 

-1. 55 10 4205 E- 03  4 . 7 7 39 370 4 E-05 

4 . 7 7 39  370 4E-05  - 1 . 44997 06 6E-03 


ITERATION  NO.  5 
SOLUTION  MATRIX 

4. 44404465 E- 02  -2.001001410257 
-2.001001410279  4 . 49 12 3144E-02 

NOT  A P.D. MATRIX 
ERROR  MATRIX 

-1. 12693011 E- 03  -5 .89318143E-05 
-5. 89 318144 E-05  -1 . 122 32356E-03 

ITERATION  NO.  6 


INSTABILITY  OCCURRED 
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